• Daniel Barrett author page, O'Reilly Media
• MediaWiki: Wikipedia and Beyond book page
• Vistaprint
• CategoryTagSorter (Vistaprint MediaWiki extension)
• Machine Learning Crash Course, Google Developers
• "Open Source Software Turns 20-Something", LinuxInsider

🕑 1 hour 4 minutes

Daren Welsh and James Montalvo are flight controllers and instructors at the Extravehicular Activity (EVA) group at the Johnson Space Center at NASA. They first set up MediaWiki for their group in 2011; since then, they have overseen the spread of MediaWiki throughout the flight operations directorate at Johnson Space Center. They have also done a significant amount of MediaWiki development, including, most recently, the creation of Meza, a Linux-based tool that allows for easy installation and maintenance of MediaWiki.

Links for some of the topics discussed:

• "What is EVA?", NASA.gov
• "Extravehicular activity" Wikipedia article
• Wikimedia Foundation blog post on NASA
• WMF YouTube video on NASA
• Summary of state of NASA wikis in 2016
• "Spacesuit Leak That Nearly Drowned Astronaut Could Have Been Avoided", Space.com, February 26, 2014
• WatchAnalytics MediaWiki extension
• Meza

The Foundation fiscal quarter running from January 2018 through March 2018 was a busy one for the Cloud Services team:

@Harej joined us as an Associate Product Manager on a six month contract. James is working on defining a product roadmap for "Toolhub". This project is an effort inspired by T115650: Create an authoritative and well promoted catalog of Wikimedia tools. You can follow his progress via the Toolhub Phabricator project and the project's pages on Meta.

@Bstorm was hired as an Operations Engineer. She joined us just before the Foundation's annual all hands event and got a crash course in the names and faces of about 300 co-workers. Brooke has a lot of prior experience in both systems administration and software development, and is coming up to speed quickly with the Cloud Services environment. Her recent projects include improvements to the Toolforge CDNJS mirror and enhancements for the automation tools we use to update the Wiki Replicas indexes and views.

Our third new team member in January was @Chicocvenancio. A long time Wikimedian and Toolforge user, Chico is working as a Technical Support contractor. If you stop by the #wikimedia-cloud irc channel and ask for !help Chico is likely to be one of the folks who tries to help you out.

@srodlund also officially became part of the team as a technical writer. Sarah had been working with us on an ad hoc basis for months, but in January we came to an agreement for her to spend 50% of her paid Foundation time working on technical writing projects. Sarah has many years of experience as both a technical writer and as a writing instructor. We are excited to have her leading our efforts to create a community of technical writing contributors for the many Wikimedia projects.

The team was also busy working on the 'normal' projects which, when things go well, are seldom noticed. Wikitech, Horizon, and the Toolforge admin console have been moved to new physical servers thanks to @Andrew. You can read more about the details in Running red-queen-style. @aborrero has been working on making software security updates easier to manage. @chasemp is progressing on our OpenStack Neutron migration project. The public facing parts of Dumps have been moved to new servers thanks to a collaboration by @madhuvishy and @ArielGlenn.

In Toolforge news, long time volunteer @zhuyifei1999 was granted Toolforge administrator rights. YiFei has been providing great technical support advice to our community and code contributions for Toolforge and related services for many months. The adminship gives him greater abilities to troubleshoot and correct problems for Toolforge tool maintainers.

• Brian Wolff MediaWiki user page
• English-language Wikinews
• Google Summer of Code
• Brian's 2010 GSoC project ("Improve metadata support for uploaded media in MediaWiki by displaying embedded IPTC and XMP metadata")
• Phan-taint-check-plugin (security testing tool)
• MediaWiki security manual
• MediaWiki's page security extension disclaimer
• EMWCon Spring 2018
• "Bashkir becomes the first language collated inside MediaWiki" (2017 Wikimedia blog post)

Photo by Alexandre Buisse (Nattfodd), CC BY-SA 4.0.

🕑 37 minutes

Bernhard Krabina is a researcher and consultant for KDZ, the Centre for Public Administration Research, a Vienna, Austria-based nonprofit that focuses on improving and modernizing technology-based solutions in government at all levels within Europe. He has been involved with MediaWiki in government for the last 10 years.

Links for some of the topics discussed:

• KDZ (English-language site)
• Verwaltungskooperation
• Datencockpit ("Data Cockpit")
• General Data Protection Regulation
• Vienna History Wiki
• patchwork.fm (Bernhard's band)

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.

If everything goes as planned, users will barely notice this change at all.

This is a lot of what our team does -- running as fast as we can just to stay in place. Software doesn't last forever -- it takes a lot of effort just to hold things together. Here are some of the problems that this rebuild is solving:

• T186288: Operating System obsolescence. Years ago, the Wikimedia Foundation Operations team resolved to move all of our infrastructure from Ubuntu to Debian Linux. Ubuntu Trusty will stop receiving security upgrades in about a year, so we have to stop using it by then. All three services (Wikitech, Horizon, Toolsadmin) were running on Ubuntu servers; Wikitech was the last of the Foundation's MediaWiki hosts to run on Ubuntu, so its upgrade should allow for all kinds of special cases to be ignored in the future.

• T98813: Keeping up with PHP and HHVM. In addition to being the last wiki on Trusty, Wikitech was also the last wiki on PHP 5. Every other wiki is using HHVM and, with the death of the old Wikitech, we can finally stop supporting PHP 5 internally. Better yet, this plays a part in unblocking the entire MediaWiki ecosystem (T172165) as newer versions of MediaWiki standardize on HHVM or PHP 7.

• T168559: Escaping failing hardware. The old Wikitech site was hosted on a machine named 'Silver'. Hardware wears out, and Silver is pretty old. The last few times I've rebooted it, it's required a bit of nudging to bring it back up. If it powered down today, it would probably come back, but it might not. As of today's switchover, that scenario won't result in weeks of Wikitech downtime.

• T169099: Tracking OpenStack upgrades. OpenStack (the software project that includes Horizon and most of our virtual machine infrastructure) releases a new version every six months. Ubuntu packages up every version with all of its dependencies, and provides a clear upgrade path between versions. Debian, for the most part, does not. The new release of Horizon is no longer deployed through an upstream package at all, but instead is a pure Python deploy starting with the raw Horizon source and requirements list, rolled into Wheels and deployed into an isolated virtual environment. It's unclear exactly how we'll transition our other OpenStack components away from Ubuntu, but this Horizon deploy provides a potential model for deploying any OpenStack project, any version, on any OS. Having done this I'm much less worried about our reliance on often-fickle upstream packagers.

• T187506: High availability. The old versions of these web services were hosted on single servers. Any maintenance or hardware downtime meant that the websites were gone for the duration. Now we have a pair of servers with a shared cache, behind a load-balancer. If either of the servers dies (or, more likely, we need to reboot one for kernel updates) the website will remain up and responsive.

Of course, having just moved wikitech to HHVM, the main Wikimedia cluster is being upgraded from HHVM to PHP 7, and Wikitech will soon follow suit. The websites look the same, but the race never ends.

🕑 1 hour 22 minutes

Mike Cariaso is the co-founder of SNPedia, a MediaWiki-based repository of genomic information (founded in 2006), and the creator of Promethease, personal genetic analysis software that uses SNPedia's data.

Links for some of the topics discussed:

• Mike Cariaso homepage
• SNPedia
• "SNPedia" article on Wikipedia
• Promethease
• ClinVar
• PubMed
• "DNA seen through the eyes of a coder"
• Semantic MediaWiki extension
• Cargo extension
• Scribunto extension

Saint Jerome Writing by Caravaggio, public domain.

🕑 39 minutes

Niklas Laxström is the creator and co-maintainer of translatewiki.net, the site where MediaWiki and most of its extensions (along with other software, like OpenStreetMap) gets translated into hundreds of languages. Niklas also works for the Wikimedia Foundation as part of the Language team, where he helps to develop code related to translation and internationalization, most notably the Translate extension.

Links for some of the topics discussed:

• Niklas Laxström user page on translatewiki.net
• Niklas's blog
• translatewiki.net
• Translate extension
• MediaWiki Language Extension Bundle
• Project Milkshake
• Tieteen Termipankki (Finnish terminology wiki)
• Elävä perintö (Finnish culture heritage wiki)
• "Multilingual Semantic MediaWiki for Finno-Ugric dictionaries" paper (see also Niklas's complete set of publications)
• "VisualEditor plugin for Translate" task

🕑 1 hour 4 minutes

Cindy Cicalese was a Principal Software Systems Engineer at MITRE for many years, where, among other things, she oversaw the creation and maintenance of over 70 MediaWiki installations, as well as development of many MediaWiki extensions. Last year she joined the Wikimedia Foundation as Product Manager for the MediaWiki Platform team.

Links for some of the topics discussed:

• Cindy Cicalese MediaWiki user page
• Wikimedia Developer Summit 2018
• EMWCon (next conference is in Houston, Texas on March 21-23!)
• SMWCon
• U.S. government monthly MediaWiki conference call
• MediaWiki Stakeholders' Group
• Multi-Content Revisions

I call this piece “Pig’s heart, with metadata”. Photo by Museum of Veterinary Anatomy FMVZ USP/Wagner Souza e Silva, CC BY-SA 4.0.

Photo by Victor Grigas, CC BY-SA 3.0.

Photo by Ralf Roletschek, Free Art License 1.3.

Long ago, the Wikimedia Operations team made the decision to phase out use of Ubuntu servers in favor of Debian. It's a long, slow process that is still ongoing, but in production Trusty is running on an ever-shrinking minority of our servers.

As Trusty becomes more of an odd duck in production, it grows harder to support in Cloud Services as well. Right now we have no planned timeline for phasing out Trusty instances (there are 238 of them!) but in anticipation of that phase-out we've now disabled creation of new Trusty VMs.

This is an extremely minor technical change (the base image is still there, just marked as 'private' in OpenStack Glance). Existing Trusty VMs are unaffected by this change, as are present ToolForge workflows.

Even though any new Trusty images represent additional technical debt, The WMCS team anticipates that there will still be occasional, niche requirements for Trusty (for example when testing behavior of those few remaining production Trusty instances, or to support software that's not yet packaged on Debian). These requests will be handled via phabricator requests and a bit of commandline magic.

Photo by Hans Hillewaert, CC BY-SA 4.0.

Photo by CEphoto, Uwe Aranas, CC BY-SA 3.0.

A Wikimedia team won the Open Minds Award in the category “Diversity” for their work with a mentoring program during the 2017 Wikimedia Hackathon in Vienna. Photo by Jean-Frédéric, CC0/public domain.

One of our quarterly goals was "Define a metric to track OpenStack system availability". Despite the weak phrasing, we elected to not only pick something to measure but also to actually measure it.

I originally proposed this goal based on the notion that VPS creation seems to break pretty often, but that I have no idea how often, or for how long. The good news is that several months ago Chase wrote a 'fullstack' testing tool that creates a VM, checks to see if comes up, makes sure that DNS and puppet work, and finally deletes the new VM. That tool is now running in an (ideally) uninterrupted loop, reporting successes and failures to graphite so that we can gather up long-term statistics about when things are working.

In addition to the fullstack test, I wrote some Prometheus tests that check whether or not individual public OpenStack APIs are responding to requests. When these services go down the fullstack test is also likely to break, but other things are affected as well: Horizon, the openstack-browser, and potentially various internal Cloud Services things like DNS updates.

All of these new stats can now be viewed on the WMCS API uptimes dashboard. The information there isn't very detailed but should be useful to the WMCS staff as we work to improve stability, and should be useful to our users when they want to answer the question "Is this broken for everyone or just for me?"

TL;DR

• Change your tools and scripts to use:
  • *.web.db.svc.eqiad.wmflabs (real-time response needed)
  • *.analytics.db.svc.eqiad.wmflabs (batch jobs; long queries)
• Replace * with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).
• The new servers do not support user created databases/tables because replication can't be guaranteed. See T156869 and below for more information. tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) will continue to support user created databases and tables.
• Report any bugs you find with these servers in Phabricator using the Data-Services tag.

Wiki Replicas

The Wiki Replicas are one of the unique services that Wikimedia Cloud Services helps make available to our communities. Wiki Replicas are real-time replicas of the production Wikimedia MediaWiki wiki databases with privacy-sensitive data removed. These databases hold copies of all the metadata about content and interactions on the wikis. You can read more about these databases on Wikitech if you are unfamiliar with their details.

The current physical servers for the <wiki>_p Wiki Replica databases are at the end of their useful life. Work started over a year ago on a project involving the DBA team and cloud-services-team to replace these aging servers (T140788). Besides being five years old, the current servers have other issues that the DBA team took this opportunity to fix:

• Data drift from production (T138967)
• No way to give different levels of service for realtime applications vs analytics queries
• No automatic failover to another server when one failed

Bigger, faster, more available

Each of the three new servers is much larger and faster than the servers they are replacing. Five years is a very long time in the world of computer hardware:

• 2 x 4 core processors (16 virtual cores total)
• 512 GB of RAM (2.5 times the old amount)
• 12 TB of SSD disks (vs 5 TB of fixed disks previously)

We have also upgraded the database software itself. The new servers are running MariaDB version 10.1. Among other improvements, this newer database software allows us to use a permissions system that is simpler and more secure for managing the large number of individual tools that are granted access to the databases.

The new servers use InnoDB tables rather than the previous TokuDB storage. TokuDB was used on the old servers as a space-saving measure, but it has also had bugs in the past that caused delays to replication. InnoDB is used widely in the Wikimedia production databases without these problems.

The new cluster is configured with automatic load balancing and failover using HAProxy. All three hosts have identical data. Currently, two of the hosts are actively accepting connections and processing queries. The third is a ready replacement for either of the others in case of unexpected failure or when we need to do maintenance on the servers themselves. As we learn more about usage and utilization on these new hosts we can change things to better support the workloads that are actually being generated. This may include setting up different query duration limits or pooling the third server to support some of the load. The main point is that the new system provides us with the ability to make these types of changes which were not possible previously.

Improved replication

The work of scrubbing private data is done on a set of servers that we call "sanitarium" hosts. The sanitarium servers receive data from the production primary servers. They then in turn act as the primary servers which are replicated to the Wiki Replica cluster. The two sanitarium servers for the new Wiki Replica cluster use row-based replication (RBR). @Marostegui explains the importance of this change and its relationship to T138967: Labs database replica drift:

[W]e are ensuring that whatever comes to those hosts (which are, in some cases, normal production slaves) is exactly what is being replicated to the [Cloud] servers. Preventing us from data drifts, as any data drift on row based replication would break replication on the [Cloud] servers. Which is bad, because they get replication broken, but at the same time is good, because it is a heads up that the data isn't exactly as we have it in core. Which allows us to maintain a sanitized and healthy dataset, avoiding all the issues we have had in the past.

The data replicated to the new servers has been completely rebuilt from scratch using the RBR method. This has fixed many replication drift problems that exist on the older servers (T138967). If your tool performs tasks where data accuracy is important (counting edits, checking if a page has been deleted, etc), you should switch to using the new servers as soon as possible.

New service names

Populating the new sanitarium servers with data was a long process (T153743), but now that it is done our three new Wiki Replica servers are ready for use. With the old setup, we asked people to use a unique hostname with each database they connected to (e.g. enwiki.labsdb). The new cluster extends this by adding using service names to separate usage by the type of queries that are being run:

• Use *.web.db.svc.eqiad.wmflabs for webservices and other tools that need to make small queries and get responses quickly.
• Use *.analytics.db.svc.eqiad.wmflabs for longer running queries that can be slower.

If you were using enwiki.labsdb you should switch to either enwiki.analytics.db.svc.eqiad.wmflabs or enwiki.web.db.svc.eqiad.wmflabs. The choice of "analytics" or "web" depends on what your tool is doing, but a good rule of thumb is that any query that routinely takes more than 10 seconds to run should probably use the "analytics" service.

Right now there is no actual difference between connecting to the "web" or "analytics" service names. As these servers get more usage and we understand the limitations they have this may change. Having a way for a user to explicitly choose between real-time responses and slower responses for more complicated queries will provide more flexibility in tuning the systems. We expect to be able to allow queries to run for a much longer time on the new analytics service names than we can on the current servers. This in turn should help people who have been struggling to gather the data needed for complex reports within the current per-request timeout limits.

A breaking change

These new servers will not allow users to create their own databases/tables co-located with the replicated content. This was a feature of the older database servers that some tools used to improve performance by making intermediate tables that could then be JOINed to other tables to produce certain results.

We looked for solutions that would allow us to replicate user created data across the three servers, but we could not come up with a solution that would guarantee success. The user created tables on the current servers are not backed up or replicated and have always carried the disclaimer that these tables may disappear at any time. With the improvements in our ability to fail over and rebalance traffic under load, it is more likely on the new cluster that these tables would randomly appear and disappear from the point of view of a given user. This kind of disruption will break tools if we allow it. It seems a safer solution for everyone to disallow the former functionality.

User created databases and tables are still supported on the tools.db.svc.eqiad.wmflabs server (also known as tools.labsdb). If you are using tables co-located on the current c1.labsdb or c3.labsdb hosts we are recommending that your tool/scripts be updated to instead keep all user managed data on tools.db.svc.eqiad.wmflabs and perform any joining of replica data and user created data in application space rather than with cross-database joins.

There will be further announcements before the old servers are completely taken offline, but tools maintainers are urged to make changes as soon as they can. The hardware for the older servers is very old and may fail in a non-recoverable way unexpectedly (T126942).

Curated datasets

There are some datasets produced by ORES, the Analytics team, or volunteers that really do need to be co-located with the wiki tables to be useful. We are looking for a solution for these datasets that will allow them to be replicated properly and available everywhere. See T173511: Implement technical details and process for "datasets_p" on wikireplica hosts for further discussion of providing some method for 'curated' datasets to be added to the new cluster.

Quarry will be switched over to use *.analytics.db.svc.eqiad.wmflabs soon. As noted previously, using the analytics service names should allow more complex queries to complete which will be a big benefit for Quarry's users who are doing analytics work. This change may however temporarily interrupt usage of some datasets that are blocked by T173511. Follow that task for more information if your work is affected.

You can help test the new servers

Before we make the new servers the default for everyone, we would like some early adopters to use them and help us find issues like:

• missing permissions
• missing views
• missing wikidb service names

To use them, change your tool to connect to:

• *.web.db.svc.eqiad.wmflabs (real-time response needed)
• *.analytics.db.svc.eqiad.wmflabs (batch jobs; long queries)

Replace the * with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).

For interactive queries, use one of:

• sql --cluster analytics <database_name>
• mysql --defaults-file=$HOME/replica.my.cnf -h <wikidb>.analytics.db.svc.eqiad.wmflabs <database_name>

Report any bugs you find with these servers and their data in Phabricator using the Data-Services tag.

Thanks

The cloud-services-team would like to especially thank @jcrespo and @Marostegui for the work that they put into designing and implementing this new cluster. Without their technical experience and time this project would never have been successful.

Learn more

• Overview of LabsDBs by @jcrespo at 2017 Wikimedia Developer Summit (video, slides)
• Data Services portal on Wikitech
• T140788: Labs databases rearchitecture (tracking)

24% of Wikipedia edits over a three month period in 2016 were completed by software hosted in Cloud Services projects. In the same time period, 3.8 billion Action API requests were made from Cloud Services. We are the newly formed Cloud Services team at the Foundation, which maintains a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement. -- https://blog.wikimedia.org/2017/09/11/introducing-wikimedia-cloud-services/

With a lot of help from @MelodyKramer and the Wikimedia-Blog team, we have published a blog post on the main Wikimedia blog. The post talks a bit about why we formed the Wikimedia Cloud Services team and what the purpose of the product rebranding we have been working on is. It also gives a shout out to a very small number of the Toolforge tools and Cloud VPS projects that the Wikimedia technical community make. I wish I could have named them all, but there are just too many!

Toolsadmin.wikimedia.org is a management interface for Toolforge users. On 2017-08-24, a new major update to the application was deployed which added support for creating new tool accounts and managing metadata associated with all tool accounts.

Under the older Wikitech based tool creation process, a tool maintainer sees this interface:

As @yuvipanda noted in T128158, this interface is rather confusing. What is a "service group?" I thought I just clicked a link that said "Create a new Tool." What are the constrains of this name and where will it be used?

With the new process on toolsadmin, the initial form includes more explanation and collects additional data:

The form labels are more consistent. Some explanation is given for how the tool's name will be used and a link is provided to additional documentation on wikitech. More information is also collected that will be used to help others understand the purpose of the tool. This information is displayed on the tool's public description page in toolsadmin:

After a tool has been created, additional information can also be supplied. This information is a superset of the data needed for the toolinfo.json standard used by Hay's Directory. All tools documented using toolsadmin are automatically published to Hay's Directory. Some of this information can also be edited collaboratively by others. A tool can also have multiple toolinfo.json entries to support tools where a suite of functionality is published under a single tool account.

The Striker project tracks bugs and feature ideas for toolsadmin. The application is written in Python3 using the Django framework. Like all Wikimedia software projects, Striker is FLOSS software and community contributions are welcome. See the project's page on wikitech for more information about contributing to the project.

Back in year zero of Wikimedia Labs, shockingly many services were confined to a single box. A server named 'virt0' hosted the Wikitech website, Keystone, Glance, Ldap, Rabbitmq, ran a puppetmaster, and did a bunch of other things.

Even after the move from the Tampa data center to Ashburn, the model remained much the same, with a whole lot of different services crowded onto a single, overworked box. Since then we've been gradually splitting out important services onto their own systems -- it takes up a bit more rack space but has made debugging and management much more straightforward.

Today I've put the final finishing touches on one of the biggest break-away services to date: The puppetmaster that manages most cloud instances is no longer running on 'labcontrol1001'; instead the puppetmaster has its own two-server cluster which does puppet and nothing else. VMs have been using the new puppetmasters for a few weeks, but I've just now finally shut down the old service on labcontrol1001 and cleaned things up.

With luck, this new setup will gain us some or all of the following advantages:

• fewer bad interactions between puppet and other cloud services: In particular, RabbitMQ (which manages most communication between openstack services) runs on labcontrol1001 and is very hungry for resources -- we're hoping it will be happier not competing with the puppetmaster for RAM.

• improved puppetmaster scalability: The new puppetmaster has a simple load-balancer that allows puppet compilations to be farmed out to additional backends when needed.

• less custom code: The new puppetmasters are managed with the same puppet classes that are used elsewhere in Wikimedia production.

Of course, many instances weren't using the puppetmaster on labcontrol1001 anyway; they use separate custom puppetmasters that run directly on cloud instances. In many ways this is better -- certainly the security model is simpler. It's likely that at some point we'll move ALL puppet hosting off of metal servers and into the cloud, at which point there will be yet another giant puppet migration. This last one went pretty well, though, so I'm much less worried about that move than I was before; and in the meantime we have a nice stable setup to keep things going.

Tool owners want to create accessible and pleasing tools. The choice of fonts has previously been difficult, because directly accessing Google's large collection of open source and freely licensed fonts required sharing personally identifiable information (PII) such as IPs, referrer headers, etc with a third-party (Google). Embedding external resources (fonts, css, javascript, images, etc) from any third-party into webpages hosted on Toolforge or other Cloud VPS projects causes a potential conflict with the Wikimedia Privacy Policy. Web browsers will attempt to load the resources automatically and this will in turn expose the user's IP address, User-Agent, and other information that is by default included in an HTTP request to the third-party. This sharing of data with a third-party is a violation of the default Privacy Policy. With explict consent Toolforge and Cloud VPS projects can collect and share some information, but it is difficult to secure that consent with respect to embedded resources.

One way to avoid embedding third-party resources is for each Tool or Cloud VPS project to store a local copy of the resource and serve it directly to the visiting user. This works well from a technical point of view, but can be a maintenance burden for the application developer. It also defeats some of the benefits of using a content distribution network (CDN) like Google fonts where commonly used resources from many applications can share a single locally cached resource in the local web browser.

Since April 2015, Toolforge has provided a mirror of the popular cdnjs library collection to help Toolforge and Cloud VPS developers avoid embedding javascript resources. We did not have a similar solution for the popular Google Fonts CDN however. To resolve this, we first checked if the font files are available via bulk download anywhere, sort of like cdnjs, but they were not. Instead, @zhuyifei1999 and @bd808 have created a reverse proxy and forked a font-searching interface to simplify finding the altered font CSS URLs. You can use these features to find and use over 800 font families.

You can use these assets in your tools now!

fontcdn

• https://tools.wmflabs.org/fontcdn/ - provides the web search interface for over 800 font families, from Open Sans to Ubuntu Mono

cdnjs

• https://tools.wmflabs.org/cdnjs/ - the full listing of over 3,000 libraries, from Angular to React

Onwiki docs

• https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#External_assets

Please give us feedback on how these features could be further improved, submit patches, or just show us how you are using them!

The shared Elasticsearch cluster hosted in Toolforge was upgraded from 2.3.5 to 5.3.2 today (T164842). This upgrade comes with a lot of breaking API changes for clients and indexes, and should have been announced in advance. @bd808 apologizes for that oversight.

The stashbot, sal, and bash tools have been fixed to work with the new version. They all mostly needed client library upgrades and minor API usage changes due to the library changes. If you are one of the few other users of this cluster and your tool is broken due to the change and you need help fixing it, open a task or better yet come to the #wikimedia-cloud Freenode channel and ask @bd808 for help.

(reposted with minor edits from https://lists.wikimedia.org/pipermail/labs-l/2017-July/005036.html)

TL;DR

• Tool Labs is being renamed to Toolforge
• The name for our OpenStack cluster is changing from Labs to Cloud VPS
• The prefered term for projects such as Toolforge and Beta-Cluster-Infrastructure running on Cloud-VPS is VPS projects
• Data Services is a new collective name for the databases, dumps, and other curated data sets managed by the cloud-services-team
• Wiki replicas is the new name for the private-information-redacted copies of Wikimedia's production wiki databases
• No domain name changes are scheduled at this time, but we control wikimediacloud.org, wmcloud.org, and toolforge.org
• The Cloud Services logo will still be the unicorn rampant on a green field surrounded by the red & blue bars of the Wikimedia Community logo
• Toolforge and Cloud VPS will have distinct images to represent them on wikitech and in other web contexts

In February when the formation of the Cloud Services team was announced there was a foreshadowing of more branding changes to come:

This new team will soon begin working on rebranding efforts intended to reduce confusion about the products they maintain. This refocus and re-branding will take time to execute, but the team is looking forward to the challenge.

In May we announced a consultation period on a straw dog proposal for the rebranding efforts. Discussion that followed both on and off wiki was used to refine the initial proposal. During the hackathon in Vienna the team started to make changes on Wikitech reflecting both the new naming and the new way that we are trying to think about the large suite of services that are offered. Starting this month, the changes that are planned (T168480) are becoming more visible in Phabricator and other locations.

It may come as a surprise to many of you on this list, but many people, even very active movement participants, do not know what Labs and Tool Labs are and how they work. The fact that the Wikimedia Foundation and volunteers collaborate to offer a public cloud computing service that is available for use by anyone who can show a reasonable benefit to the movement is a surprise to many. When we made the internal pitch at the Foundation to form the Cloud Services team, the core of our arguments were the "Labs labs labs" problem and this larger lack of awareness for our Labs OpenStack cluster and the Tool Labs shared hosting/platform as a service product.

The use of the term 'labs' in regards to multiple related-but-distinct products, and the natural tendency to shorten often used names, leads to ambiguity and confusion. Additionally the term 'labs' itself commonly refers to 'experimental projects' when applied to software; the OpenStack cloud and the tools hosting environments maintained by WMCS have been viable customer facing projects for a long time. Both environments host projects with varying levels of maturity, but the collective group of projects should not be considered experimental or inconsequential.

Debian Stretch was officially released on Saturday[1], and I've built a new Stretch base image for VPS use in the WMF cloud. All projects should now see an image type of 'debian-9.0-stretch' available when creating new instances.

Puppet will set up new Stretch instances just fine, and we've tested and tuned up several of the most frequently-used optional puppet classes so that they apply properly on Stretch. Stretch is /fairly/ similar to Jessie, so I'd expect most puppet classes that apply properly on Jessie to work on Stretch as well, but I'm always interested in the exceptions -- If you find one, please open a phabricator ticket.

The WMF and the Cloud team is committed to long-term support of this distribution. If you are starting a new project or rebuilding a VM you should start with Stretch to ensure the longest possible life for your work.

Back in the dark ages of Labs, all instance puppet configuration was handled using the puppet ldap backend. Each instance had a big record in ldap that handled DNS, puppet classes, puppet variables, etc. It was a bit clunky, but this monolithic setup allowed @yuvipanda to throw together a simple but very useful tool, 'watroles'. Watroles answered two questions:

1. What puppet roles and classes are applied to a given instance?
2. What instances use a given puppet class or role?

#2 turned out to be especially important -- basically any time an Op merged a patch changing a puppet role, they could look at watroles to get a quick list of all the instances that were going to break. Watroles was an essential tool for keeping VMs properly puppetized during code refactors and other updates.

Alas, the puppet ldap backend fell into disrepair. Puppetlabs stopped maintaining it, and Labs VMs were left out of more and more fancy puppet features because those features were left out of ldap. So... we switched to a custom API-based puppet backend, one that talks to Horizon and generally makes VM puppet config more structured and easier to handle (as well as supporting project-wide and prefix-wide puppet config for large-scale projects.)

That change broke Watroles, and the tool became increasingly inaccurate as instances migrated off of ldap, and eventually it was turned off entirely. A dark age followed, in which puppet code changes required as much faith as skill.

Today, at last, we have a replacement. I added a bunch of additional general-purpose queries to our puppet configuration API, and we've added pages to the OpenStack Browser to display those queries and answer both of our previous questions, with bonus information as well:

1. What puppet roles and classes are applied to a given instance?
2. What instances, prefixes, or projects use a given puppet class or role?
3. Which puppet classes are currently in use on Cloud VMs?

The data on those pages is cached and updated every 20 minutes, so won't update instantly when a config is changed, but should nonetheless provide all the information needed for proper testing of new code changes.

The first very visible step in the plan to rename things away from the term 'labs' happened around 2017-06-05 15:00Z when IRC admins made the #wikimedia-labs irc channel on Freenode invite-only and setup an automatic redirect to the new #wikimedia-cloud channel.

If you were running a bot in the old channel or using an IRC bouncer like ZNC with "sticky" channel subscriptions you may need to make some manual adjustments to your configuration.

See also

• T166420: Change Cloud Services irc channels for rebranding

The v0.37 build of rOSTW operations-software-tools-webservice has been deployed to Toolforge hosts and Tools-Kubernetes Docker images.

This release contains the following changes:

• rOSTWe653167c0cfd: Always cleanup manifest on stop for T163355: webservice stop says service not running but service.manifest not cleared
• rOSTW7e8da902d4ac: Sort backend provided types when displaying help

Kubernetes webservices will need to be restarted to pick up the new Docker images with the updated package installed. The new Docker images also contain the latest packages from the upstream apt repositories which may provide some minor bug fixes. We are not currently tracking the exact versions of all installed packages, so we cannot provide a detailed list of the changes.

When @Ryan_Lane first built OpenStackManager and Wikitech, one of the first features he added was an interface to setup project-wide sudo policies via ldap.

I've basically never thought about it, and assumed that no one was using it. A few months ago various Labs people were discussing sudo policies and it turned out that we all totally misunderstood how they worked, thinking that they derived from Keystone roles rather than from a custom per-project setup. I immediately declared "No one is using this, we should just rip out all that code" and then ran a report to prove my point... and I turned out to be WRONG. There are a whole lot of different custom sudo policies set up in a whole lot of different projects.

So... rather than ripping out the code, I've implemented a new sudo interface that runs in Horizon. [T162097] It is a bit slow, and only slightly easier to use than the old OpenStackManager interface, but it gets us one step closer to moving all PVS user interfaces to Horizon. [T161553]

For the moment, users can edit the same policies either on Horizon or on Wikitech. If I don't get complaints then I'll remove the UI from wikitech in a few weeks.

For nearly a year, Horizon has supported instance management. It is altogether a better tool than the Special:NovaInstance page on Wikitech -- Horizon provides more useful status information for VMs, and has much better configuration management (for example changing security groups for already-running instances.)

So... I've just now removed the sidebar 'Manage Instances' link on Wikitech, and will shortly be disabling the Special:NovaInstance page as well. This is one more (small) step down the road to standardizing on Horizon as the one-stop OpenStack management tool.

I've just installed a new public base image, ' debian-9.0-stretch (experimental)' and made it available for all projects. It should appear in the standard 'Source' UI in Horizon any time you create a new VM.

Please take heed of the word 'experimental' in the title. Stretch is not yet an official Debian release, and may still contain unexpected bugs and security issues. Additionally, the operations puppet repo has only begun to be Stretch-aware, so many roles and classes will likely fail or engage in otherwise undefined behavior.

So...

• Please DO use this image to test and update puppet classes, and to get a preview of things to come.

• Please DO NOT use this image to build any instances that you plan to expose to the public, or that you want to keep around for more than a few months.

When Stretch is officially released and we've ironed out some more puppet issues I will delete this base image and purge those VMs that were built from it in order to avoid the debt of having to support 'experimental' VMs for eternity.

Andrew will be upgrading our Openstack install from version 'Kilo' to version 'Liberty' on Tuesday the 2nd. The upgrade is scheduled to take up to three hours. Here's what to expect:

• Tools services and existing Labs uses should be unaffected apart from brief ( < 1 minute) interruptions in network service.

• Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled for most of the upgrade window.

• Creation/Deletion of new instances will be disabled for most of the window.

• Wikitech and Horizon may error out occasionally and/or display inconsistent information. Users may need to refresh their web logins after the upgrade.

Apart from fixing a few seldom-seen bugs, this upgrade shouldn't result in noticeable changes for Labs users. It will lay the groundwork for an upcoming Horizon upgrade, but that will be announced in future posts/emails.

If you are exclusively a user of tool labs, you can ignore this post. If you use or administer another labs project, this REQUIRES ACTION ON YOUR PART.

We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

VM Purge 2016

Associated wmflabs.org domains are included to identify projects by offered services.

We are not investigating why projects are needed at this time. If one person votes to preserve then we will do so in this round of cleanup.

In a month, projects and associated instances not claimed will be suspended or shutdown. A month later if no one complains these projects will be deleted.

The Kubernetes ('k8s') backend for Tool Labs webservices is open to
beta testers from the community as a replacment for Grid Engine
(qsub/jsub).

We have focused on providing support for PHP webservices with other
job types to follow. No action is required for users who do not want
to change at this time.

Advantages:

• Debian Jessie
• PHP (5.6)
• Better isolation between tools
• Testing the future platform and helping Tool Labs mature

How to switch an existing webervice to the Kubernetes backend:

webservice --backend=gridengine stop
webservice --backend=kubernetes start

How to switch an existing webservice back to Grid Engine:

webservice --backend=kubernetes stop
webservice --backend=gridengine start

If you encounter issues, please let us know at
https://phabricator.wikimedia.org/T139107 or on irc at
Cloud-Services.

For instructions and more information visit
https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes

The Wikimedia Legal team is interested in revising, updating, and clarifying the existing Labs Terms of Use governing developers and their projects on labs.

We have opened up Round 1 of our community consultation to hear feedback on the Labs Terms of Use. We will try to respond the best we can, but the main purpose of this round is to hear all your thoughts. After the feedback round, we will prepare a draft revision of the Terms based on that feedback and other minor revisions to clarify statements in existing the Terms. We will then engage in a community discussion about the revised Terms.

We plan to leave the discussion open until June 9, 2016.

horizon (OpenStack Dashboard) is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services.

Horizon requires 2fa to manage project resources. 2fa (which has long been required of all wikitech admins) can be setup on your special preferences page.

tools-login.wmflabs.org is on a new bastion host with twice
the RAM and CPU of the old one. This should hopefully provide a better
bandaid against it getting overloaded up. More discussion about a
longer term solution at https://phabricator.wikimedia.org/T131541

You can find new fingerprints at
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-login.wmflabs.org

Apologies for the interruption! The old bastion will continue to be available for now at tools-bastion-05.eqiad.wmflabs.

On Tuesday, 2016-04-05, we'll be upgrading Kubernetes to 1.2 and using
a different deployment method as well. While this should have no user
facing impact (ideally!) the following things might be flaky for a
period of time on that day:

1. The Gerrit IRC bot
2. NAGF (tools.wmflabs.org/nagf)
3. PAWS (paws.wmflabs.org, for those using it)

You can follow the upgrade at
https://phabricator.wikimedia.org/T130972 if you're interested.

In the previous part of this tutorial, we walked through how to make a very basic version of a ToDo app using Wikimedia’s OOjs UI library. Now it’s time to add a way to store and display information from our items.

Displaying info

Let’s first create a way to view the information we have about our items. We’ll start by adding a simple label to our page:

$( document ).ready( function () {
	var input = new OO.ui.TextInputWidget( {
			placeholder: 'ToDo item',
            classes: [ 'todo-input' ]
		} ),
		list = new OO.ui.SelectWidget( {
            classes: [ 'todo-list' ]
        } ),
		info = new OO.ui.LabelWidget( {
			label: 'Information',
            classes: [ 'todo-info' ]
        } );

	// ... code ...

	// Append the app widgets
	$( '.wrapper' ).append(
		input.$element,
		list.$element,
		info.$element
	);
} );

Once again, we’re adding a widget, and appending its $element to the DOM. Now we can use it to display the information stored in our widget. The ToDo items all live inside an OO.ui.SelectWidget which emits a ‘choose’ event when an element is clicked or chosen, with the reference to the chosen item as the parameter. We’ll attach a listener to this event.

$( document ).ready( function () {
	// ... code ...
	list.on( 'choose', function ( item ) {
		info.setLabel( item.getData() );
	} );
	// ... code ...	
} );

Now we have a very simple way of presenting the data stored in our item. This is a good start, but it doesn’t yet seem to be all that helpful, because the data stored in each of our items is the same as its label, and doesn’t quite give us any useful information. Let’s change that now.

Creating a custom item widget

In order to expand the functionality of the OO.ui.OptionWidget so we can store more information, we need to create our own class and extend OO.ui.OptionWidget.

Create a new file in your assets/ directory, called assets/ToDoItemWidget.js. In it, we are creating our new class:

var ToDoItemWidget = function ( config ) {
	// Configuration object is optional
	config = config || {};

	// Call parent constructor
	ToDoItemWidget.parent.call( this, config );
};
// Inheritance
OO.inheritClass( ToDoItemWidget, OO.ui.OptionWidget );

Our new function extends OO.ui.OptionWidget, by declaring OO.inheritClass( ToDoItemWidget, OO.ui.OptionWidget ); and by calling the parent constructor in the new class’ constructor.

Before we start actually defining members of this class, let’s make sure we can use it in our code. Let’s reopen index.html again.

First, add the file:

Second, change the code to use the new ToDoItemWidget class:

$( document ).ready( function () {
	// ... code ...

	// Respond to 'enter' keypress
	input.on( 'enter', function () {
		// ... code ...

		// Add the item
		list.addItems( [
			new ToDoItemWidget( {
				data: input.getValue(),
				label: input.getValue()
			} )
		] );
	} );

	// ... code ...
} );

You can try out your app again, but nothing should be different just yet. We can now start developing our new class to add the functionality we want to have.

Adding functionality

Let’s add a property that stores the creation time of our todo item.

var ToDoItemWidget = function ( config ) {
	config = config || {};

	// Call parent constructor
	ToDoItemWidget.parent.call( this, config );

	this.creationTime = config.creationTime;
};

OO.inheritClass( ToDoItemWidget, OO.ui.OptionWidget );

ToDoItemWidget.prototype.getCreationTime = function () {
	return this.creationTime;
};

Now we can set this when we instantiate the object in our initialization script, and change the info label to display this new property.

$( document ).ready( function () {
	// ... code ...

	input.on( 'enter', function () {
		// ... code ...

		// Add the item
		list.addItems( [
			new ToDoItemWidget( {
				data: input.getValue(),
				label: input.getValue(),
				creationTime: Date.now()
			} )
		] );
	} );

	// ... code ...

	list.on( 'choose', function ( item ) {
		info.setLabel( item.getData() + ' (' + item.getCreationTime() + ')' );
	} );

	// ... code ...
} );

Okay, that works, but since Date.now() is a UNIX timestamp, it looks weird. Let’s make a better method to display it.

In ToDoItemWidget.js:

ToDoItemWidget.prototype.getPrettyCreationTime = function () {
	var date = new Date( this.creationTime ),
		monthNames = [
			'Jan',
			'Feb',
			'Mar',
			'Apr',
			'May',
			'Jun',
			'Jul',
			'Aug',
			'Sep',
			'Oct',
			'Nov',
			'Dec'
		];

	return [
		date.getDate(),
		monthNames[ date.getMonth() ],
		date.getFullYear(),
		date.getHours() + ':' + date.getMinutes()
	].join( ' ' );
};

In index.html:

$( document ).ready( function () {
	// ... code ...

	list.on( 'choose', function ( item ) {
		info.setLabel( item.getData() + ' (' + item.getCreationTime() + ')' );
	} );

	// ... code ...
} );

And now it looks much better.

But we’re not done. We’ve already enhanced the items, so let’s add some real functionality in there.

Adding a ‘delete’ button to items

Another of OOjs UI’s concepts is componentization – the ability to create bigger widgets from smaller ones. This can be pretty powerful and allow for advanced functionality.

We’re going to start small. Let’s add a ‘delete’ button to our list of items. You can read about what OO.ui.ButtonWidget expects in its configuration options in the official documentation.

In our ToDoItemWidget.js we’ll add a button:

var ToDoItemWidget = function ( config ) {
	// ... code ...

	this.deleteButton = new OO.ui.ButtonWidget( {
		label: 'Delete'
	} );

	this.$element.append( this.deleteButton.$element );
};

Just like any other widget in OOjs UI, OO.ui.ButtonWidget has the $element property that contains its jQuery object. We’re attaching that to our own widget.

If you look at your app now, though, you’ll see that the button appears under the label. That’s because we need to add styles. Since we’re building our own widget, let’s do things properly and add a standalone style for it that we can then add and tweak in our CSS rules.

var ToDoItemWidget = function ( config ) {
	// ... code ...

	this.deleteButton = new OO.ui.ButtonWidget( {
		label: 'Delete'
	} );

	this.$element
		.addClass( 'todo-itemWidget' )
		.append( this.deleteButton.$element );
};

And the CSS rules:

.todo-itemWidget {
	display: table-row;
}

.todo-itemWidget.oo-ui-optionWidget.oo-ui-labelElement > .oo-ui-labelElement-label {
	display: table-cell;
	width: 100%;
	padding: 0.5em;
}

.todo-itemWidget .oo-ui-buttonWidget {
	display: table-cell;
}

There, that looks better. Now, let’s add functionality to this button.

Aggregating events

One of the best things about using a OO.ui.SelectWidget for our list, is that it uses the OO.ui.mixin.GroupElement that allows for really cool operations on a group of items.

One of those operations is an aggregation of events.

In effect, we can have each of our items emit a certain event, and have our list aggregate all of those events and respond whenever any of its items have emitted it. This means our logic can live “up” in the parent widget, consolidating our work with our items.

This means, however, that we will need to enhance our list object. We are going to do exactly what we did for our items (by creating the ToDoItemWidget class) but with a new ToDoListWidget class that extends OO.ui.SelectWidget.

Creating a ToDoListWidget

The new file ToDoListWidget.js:

var ToDoListWidget = function ToDoListWidget( config ) {
	config = config || {};

	// Call parent constructor
	ToDoListWidget.parent.call( this, config );
};

OO.inheritClass( ToDoListWidget, OO.ui.SelectWidget );

Attach it to index.html and change the initialization code:

Responding to aggregation of events

Now that we have our ToDoListWidget we can aggregate its item events.

So how is it actually done? First, let’s have our button emit a “delete” event for our item:

var ToDoItemWidget = function ( config ) {
	// ... code ...

	this.deleteButton.connect( this, { click: 'onDeleteButtonClick' } );

	// ... code ...
};

// ... code ...

ToDoItemWidget.prototype.onDeleteButtonClick = function () {
	this.emit( 'delete' );
};

Notice that this time, we didn’t use .on( ... ) but rather .connect( this, { ... } ) – this is because we are now connecting the object we are currently “in” the context of, to the event. I’ve used “on” before when we were in the general initialization script, and had no context to give the event emitter.

The string ‘onDeleteButtonClick’ refers to the method of the same name. When ‘click’ is emitted from that button, that method is invoked. It, in turn, will emit “delete” event.

Now, we need to make sure that the list is listening to this event from all of its sub-items. We will first aggregate the event and then listen to the aggregated event and respond to it:

var ToDoListWidget = function ToDoListWidget( config ) {
	// ... code ...

	this.aggregate( {
		delete: 'itemDelete'
	} );

	this.connect( this, { itemDelete: 'onItemDelete' } );
};

OO.inheritClass( ToDoListWidget, OO.ui.SelectWidget );

ToDoListWidget.prototype.onItemDelete = function ( itemWidget ) {
	this.removeItems( [ itemWidget ] );
};

We’ve used this.aggregate() to tell the group which events to listen to in its items, and we’ve used this.connect( this, { ... } ); to connect our own object to the event we aggregated.

Then, the responding method (onItemDelete) removes the item from the list.

You can now add and remove items from your ToDo app, yay!

The complete code

$( document ).ready( function () {
	var input = new OO.ui.TextInputWidget( {
			placeholder: 'ToDo item',
			classes: [ 'todo-input' ]
		} ),
		list = new ToDoListWidget( {
			classes: [ 'todo-list' ]
		} ),
		info = new OO.ui.LabelWidget( {
			label: 'Information',
			classes: [ 'todo-info' ]
		} );

	// Respond to 'enter' keypress
	input.on( 'enter', function () {
		// Check for duplicates
		if ( list.getItemFromData( input.getValue() ) ) {
			input.$element.addClass( 'todo-error' );
			return;
		}
		input.$element.removeClass( 'todo-error' );

		// Add the item
		list.addItems( [
			new ToDoItemWidget( {
				data: input.getValue(),
				label: input.getValue(),
				creationTime: Date.now()
			} )
		] );
	} );

	list.on( 'choose', function ( item ) {
		info.setLabel( item.getData() + ' (' + item.getPrettyCreationTime() + ')' );
	} );

	// Append the app widgets
	$( '.wrapper' ).append(
		input.$element,
		list.$element,
		info.$element
	);
} );

Future development and tutorials

Now that the app has basic operations, we can call this part of the tutorial over. I hope that you got a good taste as to what OOjs UI is like, and the potential it holds in quickly – but efficiently – developing JavaScript apps.

If you have any questions, concerns or corrections, please let me know in the comments. The entire code for this demo – and the text of the tutorials – is available in a GitHub repo, and pull requests are absolutely welcome.

In the next tutorials I will focus on the more advanced features of OOjs UI, like dialogs, advanced button-sets and working with a view-model to help us manage the operations in the background.

Until next time, JavaScript responsibly.

The post Tutorial: Building a ToDo app with OOjs-UI (Part 2) appeared first on .

In this post we’ll walk through creating a simple ToDo JavaScript app with the OOjs UI library, which was created by the Wikimedia Foundation. OOjs UI has a lot of power under the hood and a lot of potential for super-powerful JavaScript applications in your browser — so we will start small and grow as we go, hopefully giving you a taste of the library and its concepts.

OOjs UI itself is licensed under MIT. The code we will create is licensed under GPLv2.

Setup and prep

First, we’ll have to get the libraries we will use: jQuery, OOjs and OOjs UI.

Libraries

In your project directory, create a assets/lib/ folder that will hold our necessary libraries:

1. You can download jQuery from the official website. We’ll be using version 1.12.1
2. OOjs and OOjs UI are available from the official repositories, and there are two ways to download the files we need (see next section).

There are two main ways to get the files for both of these libraries, depending on how comfortable you are with development environment and working with git.

Getting OOjs and OOjs UI from the git repo

If you’re comfortable with git and gruntjs, this is the best way to work with OOjs and OOjs UI library files, as those will give you the most updated files each time:

• Clone the repository
  • OOjs – git clone git@github.com:wikimedia/oojs.git
  • OOjs UI – git clone git@github.com:wikimedia/oojs-ui.git
• Run npm install in each of the repositories to install the necessary packages.
• Run grunt build in each of the repositories. This will populate their dist/ folders, which is where you get the library files to use.
  • For OOjs, we will use the oojs.jquery.js file (place that one in your assets/lib folder)
  • OOjs UI is a robust library with the option of separating modules. We don’t need most of the files in the dist/ folder, all we need are these files to be copied files into your assets/lib/ooui/ folder:
    • dist/oojs-ui.min.js
    • dist/oojs-ui-apex.min.js
    • dist/oojs-ui-apex.css
    • dist/themes/apex

Getting OOjs and OOjs UI from the demo zip

If you don’t want to mess with git and grunt, you can download the demo files and extract the necessary library files directly from it, trusting that I did the job for you.

Download the zip file here.

The only caveat here is that the demo zip will likely not be updated as often as OOjs UI is updated, so be advised that you may be using an older library version.

Project files

We will start with two files for our project – the JavaScript initialization file and the CSS file.

• todo.css – Create an empty file todo.css and place it in the main directory – we will use this file later for all of our custom CSS styling.
• assets/init.js – Create a new empty file init.js and place it in the assets/ directory. This will be our initialization script.

Add to the project

Next, we will attach those files to our html page. This is how our basic page should look like now:

	
		
Demo ToDo app with OOjs UI
		

	

We will use the wrapper div element to inject our application into.

Building the base

So now that we have our basic page, we need to start writing code. Our ToDo app should have two main pieces to start with: An input to add a new item, and a list displaying all items that have been added. Since a ToDo list allows us to show a list of items that can be selected, the best starting point for this is an OO.ui.SelectWidget — so that’s what we’ll start with. You can see a demo of all OOjs UI widgets in the official demo page.

$( document ).ready( function () {
	var input = new OO.ui.TextInputWiget(),
		list = new OO.ui.SelectWidget();

	// Append to the wrapper
	$( '.wrapper' ).append(
		input.$element,
		list.$element
	);
} );

Let’s break this up and see what we did there.

One of OOjs UI’s principles is to separate the data from the UI, so each one of the widgets we’re creating is first and foremost an object that is separate from the DOM. That object contains the DOM element itself in the $element property, which we use to attach to the document, but the behavior itself (as we will soon see) is done through the general OOjs UI object.

So in short, we created two widgets — a text input and a select widget — and then attached their $element to the document. If you load your page, it should have the title and an input. The list is invisible because we don’t have a way to add elements to it yet — so let’s do that now.

Adding items to the list

We have our input, and we have the list, and now we need to connect them. OO.ui.TextInputWidget emits several events – one of them is simply “enter” when the enter key is pressed (You can see all events in the documentation). Let’s make our input add an item to the list when we hit the “enter” key.

Since the list is an OO.ui.SelectWidget we should add into it an OO.ui.OptionWidget.

// Respond to 'enter' keypress
input.on( 'enter', function () {
	// Add the item
	list.addItems( [
		new OO.ui.OptionWidget( {
			data: input.getValue(),
			label: input.getValue()
		} )
	] );
} );

That would add an item to the list. But what if we are trying to add an item that already exists? Let’s add a condition that checks whether the item exists first before it is added:

// Respond to 'enter' keypress
input.on( 'enter', function () {
	// Check for duplicates
	if ( list.getItemFromData( input.getValue() ) ) {
		input.$element.addClass( 'todo-error' );
		return;
	}
	input.$element.removeClass( 'todo-error' );

	// Add the item
	list.addItems( [
		new OO.ui.OptionWidget( {
			data: input.getValue(),
			label: input.getValue()
		} )
	] );
} );

We now are able to only add unique items to this list. When an item that already exists is added, we attach the class “todo-error” to the input. For it to actually show something, we need to define it in our CSS file. Add this to your todo.css file:

.todo-error input {
	background-color: #FF9696;
}

Now let’s try our new app:

It works! Now, let’s add a little bit of extra flair to the app.

Custom styling

Let’s make sure that our list and our input are styled a bit better, and add a placeholder to the input. Let’s go back to the piece in our code where we created the widgets, and add configuration to both:

$( document ).ready( function () {
	var input = new OO.ui.TextInputWiget( {
			placeholder: 'ToDo item',
			classes: [ 'todo-input' ]
		} ),
		list = new OO.ui.SelectWidget( {
			classes: [ 'todo-list' ]
		} );

	// code continues...
} );

The above configuration adds CSS classes to the widgets and a placeholder text to the text widget. We can now go edit our todo.css stylesheet, and add styles. Notice that we can also style the underlying objects, which (for now) we will do by calling their oo-ui-style class:

.wrapper {
	width: 800px;
	margin-left: auto;
	margin-right: auto;
}

.todo-list {
	border: 1px solid #333333;
}

.todo-list .oo-ui-optionWidget:not(:last-child) {
	border-bottom: 1px solid #666666;
}

That’s it. Now reload your app and look at your wonderful result.

The complete code

We now have a basic ToDo app. Yay! Here’s the full code of our init.js

$( document ).ready( function () {
	var input = new OO.ui.TextInputWidget( {
			placeholder: 'ToDo item',
			classes: [ 'todo-input' ]
		} ),
		list = new OO.ui.SelectWidget( {
			classes: [ 'todo-list' ]
		} );

	// Respond to 'enter' keypress
	input.on( 'enter', function () {
		// Check for duplicates
		if ( list.getItemFromData( input.getValue() ) ) {
			input.$element.addClass( 'todo-error' );
			return;
		}
		input.$element.removeClass( 'todo-error' );

		// Add the item
		list.addItems( [
			new OO.ui.OptionWidget( {
				data: input.getValue(),
				label: input.getValue()
			} )
		] );
	} );

	// Append the app widgets
	$( '.wrapper' ).append(
		input.$element,
		list.$element
	);
} );

In the next part, we’ll see how we can add more functionality, like storing and reading specific data on each object.

Go to part 2 of this tutorial

The post Tutorial: Building a ToDo app with OOjs-UI (Part 1) appeared first on .

I released the Cargo extension two months ago, though I’m only blogging about it now. (Though I did post about it to mailing lists and so on.) But I also wanted to wait a little while before fully announcing it, because on first release it was somewhat more experimental than it is now, and I wasn’t entirely sure that it would really work at all. But now I can say that it does indeed have users, and it does seem to work without any major security flaws.

Cargo is – though it still feels awkward to say it – intended as an alternative to Semantic MediaWiki. And not just to SMW itself, but to the set of libraries that SMW makes use of, and to many of the extensions that have been built on top of SMW, including Semantic Result Formats and Semantic Drilldown (though not Semantic Forms). All in all, Cargo is meant to serve as a substitute for a group of around 15 MediaWiki extensions; a more complete explanation can be found here.

Cargo can take the place of all of these extensions together, with a significantly smaller set of code, because it has a simplified approach to data storage. Instead of storing data as triples (in standard Semantic Web style), it stores data directly in database tables – it constructs a separate table for each set of data, then allows, more or less, direct SQL “SELECT” calls on those data sets. MediaWiki templates are used to define and store the data – much in the same way that they’re already used for data storage in SMW, though with Cargo it’s done more formally.

Altogether, this approach means that no custom storage or querying mechanism really had to be built for Cargo, unlike with Semantic MediaWiki; instead, the code for storing and querying data is a relatively thin wrapper around SQL, though with enough code to handle security concerns and data structures that are not well supported in most database systems, like fields that hold an array of values.

There are other interesting aspects to Cargo, and the whole issue of storing data in free-form triples vs. in structured tables is a very interesting one – really a philosophical issue. But I’m not going to get into any of that here; there’s a little more about it on the Cargo page. I do want to clarify what this means for WikiWorks. We are a full-service MediaWiki consulting company, which means that we support all manner of MediaWiki customizations and extensions; I look forward to providing support for Semantic MediaWiki installations for a long time. For new MediaWiki installations, I expect that we will be recommending Cargo if there’s ever a need for a data storage component – because it’s easier to set up, use, and maintain than SMW, in my opinion. But the jury’s still out, as they say, on Cargo, and I look forward to seeing what happens with both extensions.

We just released the MultimediaPlayer extension, which plays a list of multimedia files. It is intended for use with multimedia items hosted by an external service – not stored in the wiki.
This is, as far as I know, a different approach to Multimedia that no extension supported. Instead of showing a bunch of multimedia items, such as YouTube video thumbnails, this extension uses one player. It also generates a (usually) text list of items. Clicking on an item loads the player with the correct item.
This presumably provides a performance boost. More importantly, it is a more elegant approach to playing a large number of files. Showing 20 video thumbnails on a page is not so good. It also allows using content from mixed hosts in a way that is pretty hidden from users. Why should the user care whether a video is hosted by YouTube or Vimeo or Uncle Timmy’s? It just needs to play. Items can come from multiple sources and can be a mix of audio and video.

Default sources

By default, the player supports files hosted by DailyMotion, Instagram, SoundCloud, YouTube, Vimeo and Vine. It plays these items by embedding each service’s player. These sources also ship with some CSS that makes the players responsive.

Add your own source

The player is customizable. Admins can add code for an external player. Then that source can be included with the “multimediaitem” parser function call just like any of the default sources.

Example

This was created for the International Anesthesia Research Society and their project openanesthesia.org. You can see it in action here. They are temporarily using a very old version of the extension but the functionality is similar. Most of the items on this page come from Libsyn, and either use Libsyn’s audio or video player.

Documentation

The player’s documentation is on the mediawiki.org web site.

Geeky programmer stuff

The source code can be browsed here.

Object Oriented Programming

As time goes on, I’m becoming more and more object-oriented oriented. I’ve never created an extra class and later regretted it (because it blocked some kind of functionality I later wanted to add) and had to remove that class and insert its code into some other class. But the reverse happens to me all the time. So I decided to skip all that and put everything in its own class. The classes used were:

1. MultimediaPlayer – This defines the player as a whole, interacting with or holding the other classes.
2. MultimediaPlayerContainer – This is the Container into which the player scripts are loaded
3. MultimediaPlayerItem – Defines an Item, created by the parser function and displayed to the user as a clickable link that loads a player into the container.
4. MultimediaPlayerSources – Really a static class that just holds the code for each known source.

The Singleton Pattern

Right now, there can only be on MultimediaPlayer per page. That’s probably not ideal and may change. But it calls for only one instance to be used. So I originally created this by instantiating the MultimediaPlayer as a global. And I knew I would have to fix that. Still, I had trouble with deciding how.

In an ideal world we would create one MultimediaPlayer instance and inject it as a dependency into MultimediaPlayerHooks::renderContainer() and MultimediaPlayerHooks::renderMultimediaItem(). But since we’re using a parser hook and tag, we’re pretty set in what those functions can take as parameters.

So I went with the Singleton pattern. This is of course controversial but it’s better than using a global. So it’s an improvement. MediaWiki core uses Singletons in a number of places so I don’t feel too guilty, and I can’t think of a better way. Any ideas? Please comment.

About a week ago we had the NYC Enterprise MediaWiki Hackathon, a two-day event that was also the first-ever enterprise MediaWiki hackathon.

What does it mean to be an enterprise MediaWiki hackathon? It means that the focus is on code that’s used by companies and other organizations that have a MediaWiki installation – as opposed to by Wikipedia and the like. In practice, that usually means MediaWiki extensions.

There have certainly been MediaWiki hackathons before – there have been about five every year since 2011 – but the focus in all of them, as far as I know, has been in some way related to Wikipedia, whether it’s the development of core MediaWiki, extensions in use on Wikipedia, tools and gadgets for Wikipedia, the visualization of Wikipedia data, etc. Which is all to the good – but there’s also a need for this other stuff.

We first discussed having an enterprise hackathon at SMWCon in Berlin, last October. There was a good amount of interest expressed at the time; and if we had had the hackathon in Europe, there probably would have been more attendees, just by the nature of things. But an event in the US was easier for me to attend and orgzine, so that’s where we did this first one. I certainly hope we can have one in Europe before too long. (We also talked about naming it a “Create Camp” instead – and there are valid arguments for calling it that – but I stuck with “hackathon” for this one just to keep things simple.)

The event was held at the NYU-Poly incubator, courtesy of Ontodia, a Semantic MediaWiki-using (and -contributing) company based there – big thanks to them.

So, how did it go? Thursday, the first day of the hackathon, coincided with an epic snowstorm that dropped a foot of snow across much of the Northeast. And Friday was Valentine’s Day. And the whole event was pretty last-minute; the dates were only set a month beforehand. So turnout was certainly curtailed; but we managed to get seven people to show up on one or both days, from New York, DC and Boston; which is not bad, I think. Nearly everyone there was a Semantic MediaWiki user, and that was a big focus of the discussion and work.

The single biggest outcome of the hackathon, in my opinion, was a set of changes to Semantic Forms that Cindy Cicalese from MITRE and I worked on, that will allow for easily displaying alias values for dropdowns, radiobuttons and the like. That’s a feature that SF users have been asking about for a long time. We also got a specific Semantic Forms implementation issue resolved; people looked into setting up a semantic triplestore (though the attempt was ultimately unsuccessful); and there were various discussions about large-scale SMW-based data architecture, skinning, and other topics.

What can we learn from all this?

• A hackathon doesn’t need to be big. More projects are of course generally better, but we managed to get a bunch of stuff done with our small size. Having a small group helped us in getting free space, which kept costs minimal. And the round-table discussions we had at the beginning, introducing ourselves and talking about the projects we wanted to see done, might have taken a lot more time with a large group. (Or simply have been prohibitive to do.)
• It’s good to have people think about what they want to work on ahead of time, and write their ideas on the wiki page. That helps organizers, and participants, try to plan projects out ahead of time, to maximize productivity.
• Not all hackathon results are just code – though I’m of two minds about this one. There were some good discussions, and probably necessary ones, about various aspects of organizational MediaWiki usage. In that way, this hackathon resembled the informal part of conferences – the discussions that happen during breaks, over lunch, etc. These are often just as important as the main part of conferences, and at a hackathon you can have those kinds of discussions in a really focused way. (This hackathon was certainly not unique in that respect.) Still, as a developer, I’m focused on creating and improving code, and that to me is the real measurable output of such events. So perhaps it’ll be a while before we know the full outcome of this hackathon. But judging from people’s feedback afterwards, even time spent not writing code was time well spent.

If you’ve read any of my previous posts, you should know by now that I have been participating in the Google Summer of Code internship, working for the Wikimedia Foundation. And now, as the summer has ended, so has this internship, and it’s time for a summary. So here it is in a crunch:

Okay, yeah, this can’t be the real summary, so let me try and summarize this tremendous summer adventure with a bit more content.

Wikimedia Foundation’s VisualEditor

This summer, I worked for the Wikimedia Foundation, which is the organization behind the MediaWiki platform that is the system behind the worldwide Wikipedia websites. More specifically, I worked with the VisualEditor team.

VisualEditor is a highly anticipated upgrade to the way Wikipedia articles can be edited. Ever since MediaWiki (and Wikipedia) was born, editing involved using a special syntax called “Wikitext”. The systax is relatively straight forward: Wrap text with ‘===’ to get a heading, or ”’ to get a bolded text. Start a line with a star (*) for a bullet list or a pound (#) for a numbered list. The idea was to help editors write articles simply without having to know the elaborate functionality of HTML tags.

And for the most part, it worked, rather beautifully, even. It’s all about accessibility; Wikipedia has evolved into the biggest encyclopedia, making it possible for anyone to edit and add to its growing body of knowledge without having any technical know-how.

But as Wikipedia grew in size and in articles became more elaborate, so did wikitext. It evolved to include templates and styles. Wikitext evolved with the community to allow for more and more possibilities and complex usage. Writing articles involved learning a new syntax, and while it was easier than the alternative HTML tags, it still has a learning curve, especially for new users to learn, and even users who are familiar with it sometimes have challenges using it properly. But the biggest issue (at least as far as I am concerned) is using it with right-to-left languages.

The issue of Right-to-Left languages

Wikitext is plain-text code that is left-to-right, so writing articles in right-to-left language means mixing directions, and that’s a recipe for trouble. Symbols like pipe “|” colon “:” or parentheses “(” and “[” flip the sentence around when the sentence has mixed directions. You can read about some of those challenges in my blog post about that.

VisualEditor: The Solution

That’s where VisualEditor comes in. Instead of having to type articles using wikitext syntax, the idea is to have a visual interface that allows anyone to add content into articles easily, like a “WYSIWYG” system or a Word processor. This serves the purpose of making it easier for everyone to edit articles, even without knowing wikitext or being intimidated by weird symbols in the text editing area — and also help right-to-left language speakers to edit articles without bidirectionality frustration. And, also, it’s a lot more convenient to edit articles the way they are supposed to look than their source code.

Project Summary: Right-to-Left Language Support in Visual Editor

So, my project involves adding in language and right-to-left support, both in terms of fixing up bugs that prevented VisualEditor from being used on right-to-left Wikis, and also developing tools to help editors handle languages when editing wiki articles.

Language Inspector

The Language Inspector adds a “Language” button to the toolbar that allows users to mark certain text as a particular language. The code uses UniversalLanguageSelector to provide visual GUI for language selection and relate the language to its directionality.

A screenshot of a draft version of the VisualEditor language inspector. Credit: Amir Aharoni

TemplateData Generator

A MediaWiki extension that provides visual GUI for editing json-based <templatedata> tags.

TemplateData is a json string that is used by VisualEditor to recognize the elements and parameters of MediaWiki templates. Since templatedata is json string, it is not easy for users to manually edit it — especially so in right-to-left environment, where the mix of english-based json with rtl-parameters can make the editing process incredibly challenging.

TemplateData Generator allows users to use the GUI to either create the templatedata json string from scratch, by importing parameters, or to edit an existing one, without having to edit the text itself. The extension is also planned to be merged into an existing TemplateData extension that is deployed on the international Wikipedia.

TemplateData Generator with Hebrew parameters

Bug fixes

On top of those two projects, I’ve also spent some time fixing up right-to-left related bugs, some of which were blocking VisualEditor from being used in right-to-left wikis:

• VisualEditor: Arrow keys move the caret in the opposite direction in RTL environment
• In RTL (hewiki and arwiki) wikis, can’t remove categories
• VisualEditor: {Ctrl,Option}+{Delete,Backspace} doesn’t delete a word in an RTL wiki 
• VisualEditor: Link inspector surface doesn’t pop up when the user language is right-to-left
• VisualEditor: CSS broken in RTL (Directionality and position fixes for dialogs and frames) 
• VisualEditor: Link inspector in a slug can select content in the previous or following paragraphs 
• VisualEditor: “[edit | edit source]” appears jumbled in RTL wiki with LTR interface 
• VisualEditor: Transclusion icon appears on the opposite side of the page in RTL wikis 
• VisualEditor: Location of the inspector menu is wrong for floated content in RTL wikis 
• VisualEditor: Autocompletion for template names doesn’t work in RTL 
• VisualEditor: By default, images should not set the alignment 
• VisualEditor: Link widget in a slug fails with “No class registered by that name: undefined” 
• VisualEditor: After adding a link, cursor appears before rather than after the new insertion 
• VisualEditor: Link inspector input text always has LTR direction 
• VisualEditor: Non-contiguous rendering of references in LTR text in RTL mode 
• Page settings dialog is not oriented for RTL 

It. Was. Awesome.

So that was the actual work, the “deliverables” that resulted from this great internship. But there were quite a lot of other beneficial results. Here’s a couple of things I’ve learned.

Programming Collaboratively

Since MediaWiki is a collaborative project, I learned a lot about how to write code with other people. Up until this project, my programming was mostly by myself. Sure, I’ve cooperated on code before, but it was mostly involving a tiny group of people, or it was me getting into someone else’s code and sending in a patch request. I knew about the idea that code can actually be worked on — and reviewed! — collaboratively, but it’s been completely different actually doing it.

I’ve learned that the industry relies on communication and cooperation to produce working programs, and while there are teams that do work on different aspects of these products, cooperation over making them work means there is “fresh eyes” going over the pieces, improving the system in general. Cooperating on code not only made the code I worked on be better, it made me be a better programmer.

Communication

I learned to communicate the problems I encounter and the solutions I come up with clearly. Sometimes, while thinking about how to explain the problem I got stuck with, I realized I might have a potential solution. Explaining to others clearly meant understanding things better for myself.

Since I speak Hebrew and my work involves Right-to-Left issues, I also tried to be in constant communication with the user base – especially the Hebrew Wikipedia – to hear about the bugs and problems that frustrate them. In that aspect, I had to learn how to translate the user complaint into a technical requirement and vise versa – when technical features came out, it was helpful to communicate them back to the users for their comments.

Programming Properly

I started this internship feeling rather insecure about my programming ability.

I’ve been programming various languages since I was rather young, but it was always more a hobby than an actual profession. I could make programs work, and for the most part, they work just fine, but this project involved getting into a really big, complex codebase and starting to make considerations that I’ve never had before. This project made me start considering not just working code solutions, but solutions that are efficient, that work with long-term product strategy, that work consistently, that survive millions of users and provide proper usability in wide variety of systems.

I wasn’t sure I can do that, but thanks to extremely talented and helpful team and mentors, and passionate users who pointed out the challenges they wanted fixed, I delved in, and I managed.

I think I did more than just managed, but the best thing of all of this, as far as I’m concerned, is that I’ve learned — a lot — about how to do things right. Really right. And I tell ya, I had a blast learning it.

Advice for future Google Summer of Code interns

If you’re going for a Google Summer of Code project, you know you’re going to work with Open Source companies, and if you really want to contribute and also get the most out of the internship, here is my two cents’ worth:

Don’t just Do it — Own it

If you want to really learn and get the best out of Google Summer of Code, you should own your project.

Find bugs that are related to the system you’re working on and fix them. Get in touch with the user base and encourage them to talk to you, be bold, ask questions, bug people on IRC and the mailing lists, try to see if there are parts of your projects that touch other related projects and go bug them, too.

Utilize the Community

I think that’s the best thing I’ve learned from this internship and the best advice I can give to future interns: Open source projects are a community.

You don’t just collaborate on code, you work together, plan together, and help one another.

Don’t be shy, don’t be mute, and don’t work in a bubble. Communicate, cooperate, discuss, do NOT let yourself be frustrated over problems for too long — the community is right there, waiting to be challenged with questions.

Do things right

Don’t just program – program with the open source code in mind. Document your code, learn the organization’s code style, ask for reviews and try to write code that other people can understand and tweak.

Don’t be intimidated. It’s worth it, really, because before you know it, people start sending others your way for answers on the things you’re working on.

And that’s the Best. Feeling. Ever.

So, yeah. It. Was. Awesome.

Thanks and Appreciation

So, as I sum up this incredible summer, I want to finish up with a thank-you list. This list is not complete, not even by a long shot; MediaWiki is a huge project with lots of volunteers, and many of them affected me and the progress of this project. But I will point out a few of the more notable groups of people who helped me this summer, and to which I am incredibly thankful.

You guys rock, y’all. You made this summer awesome, left to right and right to left!

So, Thank You to —

• My mentors, Amir Aharoni and Inez Korczyński, for being incredibly accessible, supportive and helpful. Thanks, Amir, for encouraging me to take this project head-on despite my doubts about my abilities, for helping me see the big picture and what I need to look for, and, most of all, for making me feel like you’re actually enjoying the fact I constantly bug you. Thanks Inez for helping me get into a rather complex code and understand it enough to work with it, and for not letting me panic when I thought I broke things. Also, for not letting me break things too badly.
• Timo Tijhof and Roan Kattouw from the VisualEditor team, for not only being experts, but allowing me to squeeze their expertise over and over again, and for being incredibly patient with my “wait wait wait, what?” baffled moments. Timo, thanks for showing me (by example and by code comments) what to look for and how to consider strategizing code, and for helping me strive to produce actually good code that others can use and work with. Roan, thank you for your endless patience (and yes, I mean e-n-d-l-e-s-s), for helping me get into the codebase, and for helping me climb out of the pits of panic when I managed to screw things up, and thank you both for making me feel like I kinda know what I’m doing, even when I didn’t.
• James Forrester, VisualEditor product manager, for alleviating my worries and insecurities by being extremely welcoming, for making me feel like part of the team by assigning me bugs, by being there when I needed help (which was quite often, really,) by inviting me over to San Francisco to meet the team, and for offering me a job. That was awesome.
• The VisualEditor superteam – Trevor Parscal, Ed Sanders, Rob Moen, and David Chan, who were there for my questions even when they didn’t realize it. Also, for an amazing San Francisco visit!
• To the Wikimedia Language Team, and their hard work, for making me realize that speaking and writing RTL does not an RTL-programmer make, and for the absolutely rawkin’ UniversalLanguageSelector.
• Sumana Harihareswara and Quim Gil without which I would not have submitted a proposal on time, or at all, for being totally welcoming for newbies like me, and for the great support throughout!
• The Hebrew Wikipedia editors and system managers and users, for complaining and complementing and adding bugs to the list, so we can deal with them and make RTL support in VisualEditor awesomer and awesomer.
• … And anyone and everyone who I interacted with this summer, from Parsoid to Vagrant to the various office hours and late-night-a-la-early-morning banter in IRC.
• And to my parents, who brought me this far (this works better in Hebrew)

More Resources

• The Wikimedia Foundation
• VisualEditor
• The Wikimedia Foundation blog: Updates from the Language Engineering Google Summer of Code projects
• My Project in the Google Summer of Code 2013 website

The post GSoC2013 Summary: Right-to-Left Support in VisualEditor appeared first on .

English is written left to right. Hebrew is written right to left. We know that. Browsers (for the most part) know that too, just like they know that the default directionality of a web page is left-to-right (LTR), and that if there is a setting that explicitly defines the direction to right-to-left, the page should flip like a mirror. Browsers are smart like that. Mostly.

But even browsers have problems when deciding what to do when languages are mixed up, and that, my friends, is a recipe for really weird issues when typing and viewing bidirectional text.

On The Bidirectionality of Characters and Strings

Before we delve into some interesting examples of mixed-up directionality problems, we should first go over how browsers consider directionality at all.

We already said that English is recognized as an “LTR” language (Left-to-Right), and Hebrew, Arabic, Urdu (and some others) as RTL languages (Right-to-Left). These are fairly clear, and if you type a string that consists of these languages on their own, the situation is more or less okay (but we’ll go over some issues with that later)

But not all characters in strings are equal.**

Hebrew and English (and a couple of others) are of the “strong” directionality types, the ones that not only have direction but also affect their surroundings. Some characters have “weak” directionality, where while they have directionality internally, they don’t affect characters around them. And some characters are merely neutral, whereas they get their directionality by their surroundings. Oh, and there are also some characters that may (and do) flip around visually depending on the text they’re in.

Don’t worry. I’m going to explain eeeeeeeeeeeeeeverything.

Almost.

Kinda.

Well, I’m going to try, so just keep reading.

Character Directionality Types

Unicode, which is the encoding system most common online, defines character type directionality for groups of characters as either “strong”, “weak” or “neutral”. These types control how these characters are presented inside a string.

Strong Types

Strong types are character sets that have explicit directionality. Hebrew is right-to-left, always. English is left-to-right, always.** When I type in either of those character-sets, my characters would appear in sequence, one after the other, according to the directionality. This is how the word “Hello” appears from left to right, while the word “שלום” appears from right to left.

Strong types also set the directionality of the space they’re in, meaning that if I inserted any characters that have weak or neutral directionality in the middle of the sentence you’re reading now (and I have already done that) they will assume the direction of the strongly typed string — in this case, English. So, strong type isn’t just about the character itself, but also its surroundings.

Weak types

Weak types are fun. These are sequences of characters that might have a direction, but it doesn’t affect their surroundings, and may be adjusted based on their surrounding text. In this group are characters like numbers, plus and minus signs, colon, comma, period and other control characters.

According to the Unicode bidirectionality algorithm specifications, weak types resolve according to the previous characters.

Neutral types

Neutral types are the funn’est. Neutral characters are character types that can be either right-to-left or left-to-right, so they completely depend on what string surrounds them. These include things like new-line characters, tabs and white-space.

According to the Unicode bidirectionality algorithm specifications, neutral types resolve their directionality according to the surrounding text.

Implicit Level Types: When what you type is not quite what you get

So we have strong types, weak types and neutral types, but that’s not where our directionality double-take ends. In fact, the real doozies are characters that are resolved differently (as in, they take literally different shapes) in either RTL or LTR.

Yes, you read that right, they actually literally and quite visibly look different when written inside an LTR string versus inside an RTL string.

The best examples for this are parentheses and (my personal best friend) the bracket. These symbols are, in fact, icons that represent direction already. The button on your keyboard that has “(” on it is not quite that, but rather a symbol of “open parentheses”. In English (which is left-to-right) the symbol is naturally ( to open parentheses, and ) to close them. But in Hebrew and Arabic and the other RTL languages, the “open parentheses” symbol is the reverse ), since the string is right to left. So this symbol would appear on your screen either ( or ) depending where you typed it.

I know, right?

Mishmashing Both Ways

In general, if one uses only one direction in a document (specifically online) the problems are not as noticeable, because the strongly typed text surrounds all other weak and implicit-level character types, making them its own type by default.

The issues come up when we have to mix languages and directions, or use RTL language inside a block that is meant for LTR. This happens a lot online — if there is no explicit dir=”rtl” anywhere in the HTML document, the document defaults to LTR directionality. The directionality of the page (either by using dir=’rtl’ or dir=’ltr’ or not using dir= attribute at all and relying on it’s default fallback to ‘LTR’) is considered to explicitly set the directionality of the expected text. So, any characters of ambiguous directionality will take on the direction that was set by that attribute.

If, say, I try to type an RTL language inside a textbox in a page that has dir=’ltr’, I can run into a lot of annoying problems with punctuation, the positions of segments of the sentence, and mixing languages of a strong type. The same happens the other way around, if I try to type an LTR language (Say, English) inside an RTL-set textbox.

It can get so confusing, that, quite often, as I try to figure out how to type LTR text into an RTL box and see how my text actually organizes itself, my state of mind is pretty much this:

The Good, The Bad, and The Ugly

So, obviously, the creation of Unicode was much superior to the reverse-typing (and the need to use multiple individual fonts) that existed before it. Browsers tend to follow the Unicode rules (though some apps that do their own rendering sometimes don’t, but that’s a different issue.) And this Unicode directionality algorithm gives us a lot of really Good Things to work with when typing different directions, but it also has so Bad Things, and occasionally, even some really Ugly Things.

The Good Things

There are, indeed, a bunch of good things that happen due to Unicode’s bidirectionality algortithm. As I’ve already mentioned, the fact RTL users can type their language normally (and not backwards) is already a good thing (and I know, I used the system when it didn’t have that nice feature.)

Some other benefits of the bidirectionality algorithm is the fact we can use numbers (which are weakly typed LTR) inside RTL text. So, for instance, consider this text:

ניפגש ב09:35 בחוף הים

Literally, this means “we will meet at 09:35 at the beach”. Notice, though, that even without any directionality fixes the numbers 09 and 35 are left-to-right as they should be, because that’s how numbers are read — but I didn’t really need to manually reverse my typing when I wrote this sentence — the browser did it for me.

Here’s a nice exercise though: select that sentence. When you do, you can see exactly what piece has what directionality.

Which leads me to–

The Bad Things

Selections

Selections are a major part of the problem of bidirectional text. As you can see from the example of the “good thing” (that I don’t need to reverse typing) there is also a bad side, which is how to select my text. Selection can be LOGICAL or VISUAL. This is also true to cursor movement, which we will go over in a second.

Visual selection is simply that — visual — which means that the selection treats the segment of text as if it’s one continuous block, regardless of directions.

Logical selection means the text is divided to its bidirectional pieces. That means that if I start my selection at the beginning of an RTL text (at the right) and drag my mouse towards its end (to the left) the selection will split when I reach the number part, because the numbers are left-to-right.

This is, indeed, logical, because it goes from logical “start” to logical “end”, and since the text is bidirectional, those two pointers are different for each of the sections. It makes a lot of sense, but it can be confusing.

Cursor Movement

Similarly, the cursor can also move either logically or visually. This can be a little confusing, and sometimes this behavior is inconsistent across platforms. Most of the time, though, the movement is logical.

So, here’s a quick test of where this behavior can become really weird. Consider the following sentence. It is inside a textbox so you can select it and move your cursor within it properly.

Try to select the text from the start (left) to the end (right). See what happens when you hover over the Hebrew words?

Now, if you move your marker inside the given textbox, the cursor (in Chrome and Firefox in Windows, at least) will move VISUALLY and not logically. That is, you can just move from end to start as if there are no two different languages there.

But try to copy/paste this string into Notepad (or equivalent simple software) and move the cursor from start to end. Usually, those editors would move the mouse LOGICALLY. Which, to be fair, makes more sense than visual movement.

It also shows you how RTL behavior can be somewhat unpredictable; some programs do it this way, some that way. Some browsers will go visual, some logical, and there are some CSS rules that can override those decisions, too, so it may change on a website to website basis.

Nice, eh?

Punctuation Marks

Well, that was a textbox that was “LTR” to begin with. What happens, though, if I write a Hebrew sentence in an LTR box, or the other way around – an English sentence in an RTL textbox? That’s when our lovely friends — the weakly typed punctuation marks – come out to play.

This is an English sentence inside an RTL textbox. The first sentence ends with a period. So is the second and the last one.

Whoops, where’s the final period?

Here’s the reverse version:

זהו משפט בעברית בתוך קופסת טקסט משמאל לימין. המשפט הראשון נגמר בנקודה. כך גם המשפטים השני והשלישי.
Where’d that final period go?

Two languages together, Koombaya

Here’s something even better, though, that relates to both selections and cursor movement (and rendering, and usage, and and — anyways).

The above examples featured some strong type (English or Hebrew) that is mixed with some weak typed (numbers) and is mixed up by the neutral type (white space). But what if I create a string that has two opposite strong types mixed with neutral type white-spaces and weak type punctuation?

Go ahead, try to select that sentence from beginning to end:

Remember that English is strongly typed for LTR but עברית is strongly typed for RTL. When mixing English ועברית together you may get some surprising results.

Or the reverse:

(Hat tip to Amir Aharoni for this one)

Let’s go over what goes on in that horrific textbox for a minute. First of all, part of the problem in the first textbox is that the textbox was forced RTL, and since most of the text in it was English, it broke in weird places. Here’s the sentence when it is forced to be LTR:

Notice, though, that the textbox problem also happened just the same in the reverse case, where the box was LTR and the sentence was mostly RTL.

With a forced-RTL textbox (and majority of text strongly typed for LTR) the spaces took the directionality of the text they were surrounded by – which is LTR. Then we had a strongly-typed RTL word in Hebrew, which made the space inside it turn RTL, but the surrounding white space (the one between the RTL word and LTR sentence) was still affected by the surrounding text – which is LTR.

If you’re still with me here, this may help drive the point home. Essentially, you had this:

The entire sentence structure was right-to-left, but the small English segment was left-to-right.
Overall “chunk” direction was RTL. Each chunk had its own internal direction. When you read it, it looks all jumbled — because it is.

And that happened exactly the same (only in reverse) in the second textbox. With LTR instead of RTL, and vice versa.

I know. I… know.

The Ugly Things

Now we move to the ugly area, the things that are not just difficult behavior, but are also producing visually different results. Remember those weak typed and implicit-level types? That’s where these come in, and they, I tell you, they have a blast confusing us thoroughly.

White Spaces

White spaces are implicit-level types, which means they are defined by the text they live in. The spaces in the sentence you’re reading right now are implicitly LTR, since they are inside an English text. The white spaces here:

במשפט הזה יש רווחים ואלה מוגדרים ימין לשמאל

Are implicitly RTL because they’re inside hebrew, even though the page itself is LTR.

This is good, but it also produces some weird results. Consider the situation where I have a set of numbers inside a text. The numbers are separated by whitespace — and the whitespace is defined by the surrounding text. But numbers themselves are “weak” typed — which means they do not affect their own surroundings (even though they are internally LTR) The whitespaces would have to take their directionality from whatever words surround the entire segment of numbers.

This sounds weird? The behavior is even weirder. See this, for instance:

I purposefully encapsulated those numbers in an RTL text, and so the whitespaces that separate these are still LTR. What do you think would happen, though, if I replace those english words with Hebrew (RTL) ones?

Well, this example is exactly the same sentence and sequence of numbers, in the same exact order, with the single difference that “Start” and “End” were replaced by their prospective Hebrew words.

The numbers are reversed! The numbers… are… Head spinning yet?

This might be weird, but it makes sense; the spaces are now encapsulated in an RTL text which means they are now RTL. The space in rtl sentences is right-to-left, so the grouping of numbers go from the right and to the left.

But I think your head isn’t spinning fast enough just yet. What would happen if we added spaces inside the number grouping itself? I mean, the numbers are internally LTR, but the space is RTL, so we will add a space to break the group and.. and the group will go… spinning?

Try it. Add a spaces to the number groups below.

See it? SEEEEEEEEEE it?

Yeah. Exactly.

Parentheses and Brackets

As we discussed earlier in this post, brackets and parentheses are, in fact, representing “start-of” and “end-of” which means that depending where they are inserted, they may appear on different directions on your screen.

So, if I press the button that has a nice little “[” on it on my keyboard (below the { and near the P) I will get different results in LTR and RTL.

This means that this code:

LTR:
<span dir="ltr">[</span>
RTL:
<span dir="rtl">[</span>

Becomes this:

LTR:
[
RTL:
[

Yes, I clicked the same button. Yes, I’m sure. You’re welcome to go over the source.

More than being a weird thing, this effect makes it incredibly frustrating when, inside an RTL textbox, there’s a need to add some html <tags>. And, yes, this happens in Wikipedia, and in the RTL Wikipedias too.

Try adding a <span style=”font-size: 2em”> to some segment of the text below. Good luck, stay sane, and remember to breathe. If you feel especially adventurous, you could also try to insert some wikitext, like a link to a page “Somewhere” (english link) with a hebrew caption.

Want to go even wilder? Add some English text after the Hebrew one, and try to set some <a href=”something.html”> </a> starting from the Hebrew string, and ending at the English one.

Type it all, don’t cheat and copy/paste. Try it for realz. Go ahead, play. Experiment. Go RTL crazy.

So how is this relevant to VisualEditor?

As a text editor, VisualEditor expects users to type into it, and that they do. They also do that in multiple languages, and, more often than not, in mixed languages inside the same article. Mixing languages is extremely common, especially in Wikipedia, when there’s a need to provide the original script of a word taken from another language, or a city name in its native script, etc.

But as we saw, typing can be tricky, especially when we mix directions. We have to make sure we allow the users to type while seeing the result they will get in the page logically. We also have to make sure that their typing makes sense, and that if there is a need to describe a specific span of text as a different direction, they can do that easily. We have to make sure their input is interpreted correctly, that RTL appears properly in the ContentEditable screen, and then renders properly in the article that is saved.

Also, as you can see from my above example with the [ character — there’s a difference between the HTML code and the resulting rendering. That is, I typed [ but got ] and [ appeared in the code, but ] appeared in my resulting rendered markup. Which should happen inside VisualEditor? WYSIWYG is quite a lot more different when what you type is expected to be flipped.

These things aren’t impossible to deal with, but they are quite challenging and they often require decision-making about what a user should expect. Most applications online (and offline) have problems dealing with LTR/RTL typing, making these strategic decisions even more complicated. The behavior needs to be designed according to what we think is the best way to do it, and not what the RTL users expect — because as you can see from the current behavior, RTL users usually expect horrendous behavior.

It’s the good kind of challenge, though. The kind a lot of people care about finding a good way to fix.

But wait, There’s more

There’s a bunch of other issues with bidirectional text, some of which are problems that exist in published software and apps online and make RTL’er’s lives rather annoying. I may write about that at some point, and share my RTL frustration.

In this article we went over issues with RTL strings inside LTR boxes, problems with characters of ambiguous directionality, with selections and cursor movements and general “huh”isms. There are, of course, more RTL hardships, but this post was meant to serve as a sort of introduction to the main and most common bidirectionality issues.

I hope you’ve enjoyed it. At least, I hope you now understand what the programmers (and RTL users!) need to deal with. I tried to make it easier on you with some animated gifs. You’re welcome.

And so, until next time: !oo-eldoot

Remark Regarding Languages and Scripts

In this article, I use the term “Language” to refer to English and Hebrew letters. In fact, I should be using the term “Script”, to refer to the letters and characters themselves. The difference comes mostly from the fact that while Hebrew and English are languages, they each use characters that may be used in other Languages. For instance, English uses latin script, and Hebrew script can be used in Yiddish language as well.

So, take into account that this is the case, and that the actual letters that are used and are LTR or RTL are really “script” and not quite the language — since the browser doesn’t really care what words you literally type using these scripts.

For the sake of simplicity and to try and reduce some of the confusion, however, I made a tactical decision to group it all up to the most familiar terminology of “Language”.

(Thanks MatmaRex for pointing out I should at least mention this difference)

Useful Links

• Unicode Bidirectional Algorithm: http://www.unicode.org/reports/tr9/
• Bidirectional Text Requirements for Visual Editor (by Amir Aharoni) https://www.mediawiki.org/wiki/Visual_editor/Bidirectional_text_requirements
• VisualEditor: Support for right-to-left / bidirectional content Tracking bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=33126
• Blast from the 90s #1: Hebrew/English web fonts for Windows 3.1
• Blast from the 90s #2: Tools to write and read Hebrew online

The animated gifs in this post are taken from www.reactiongifs.us

The post The Language Double-Take: Dealing with Bidirectional Text (or: Wait, ?tahW) appeared first on .

I sit in front of my trusty computer, coding-away right-to-left popup-location fixes in anticipation of the new VisualEditor deployment in the Hebrew Wikipedia. The hard part, I tell myself (and with good reason) is calculating the mirroring coordinates; which object to I use as parent to mirror my coordinates against?

As I’ve explained in a previous post, the positioning and mirroring of nested elements can be a real challenge — the RTL/LTR Vertigo.

This time, the element whose position I was trying to mirror popped up inside another widget — a frame inside a frame — which made positioning all that much trickier, since it’s all relative to one another. In cases like these, I find myself following up on the relative positioning of several elements in the nesting chain; even Einstein’s head would’ve spun.

The principle, however, is the same — flip the ‘left’ position with the mirrored ‘right’ position, relative to the parent container.

Similarly to last time, the fixed code looked like this:

// ('dimensions' variable was set above according to ltr positions)
// Fix for RTL 
if ( this.$.css( 'direction' ) === 'rtl' ) {
	dimensions.right = parseFloat( this.$.parent().css( 'left' ) ) - dimensions.width - dimensions.left;
	// Erase the value for 'left':
	delete dimensions.left;
}
this.$.css( dimensions );

In VisualEditor, this.$ represents the jQuery object of the object we’re in (‘this’). Since I am calling for a jQuery function ‘css()’, I must refer to the jQuery object, rather than the VisualEditor object.

I can test the RTL direction property this.$.css( ‘direction’ ) on my popup widget because the direction is an inherited property; even if I didn’t set “direction:rtl” explicitly, the element will inherit it from the parent, who will inherit it from its parent.. etc etc forever and ever. Well, at least until an element with defined directionality is found, and if none is found, then, as far as it should go, the default is LTR.

So this should have worked. And in Firefox, it did. But not in Chrome.

Firefox and Chrome

The coordinate mirroring worked beautifully in Firefox.

It also failed beautifully in Chrome. But it’s all about math, isn’t it? Parent position minus some left value, minus some width minus– math is math. Is math. So why, I begged my screen. “Whyyyy aren’t you working??”

The Real Problem

The problem with Chrome wasn’t the cause, it was a symptom. The real problem was the way I read through the positioning of the elements. In specific, the problem was this line:

this.$.parent().css( 'left' )

This line reads the parent object’s css “left” property, which was defined (I checked!)  so it should have worked. In theory, I should have gotten the ‘left’ position of the parent. Right?

Sure. I did get the left property — but that’s exactly the problem. What I got was the CSS property of the element, I didn’t really get the actual coordinates the element was positioned. Not quite, not exactly, and not entirely.

css(‘left’) vs position().left vs offset().left

There are several ways to read the position of an element, and each one of those gives you a slightly different value. They’re all true, it just depends what, exactly, you mean to use.

css( ‘left’ ) returns the calculated value of the CSS property. It is the “left:10px” that exists either in the css stylesheet or in the style=” property of the element.

position().left returns the position of the element relative to the offset parent.

offset().left returns the position of the element relative to the document.

Differences in Browsers and Screens

There may be subtle differences in the way browsers render result differently, the value of the css(‘left’) property can be different across browsers. position() and offset() are actual x/y values, so requesting these values will give me the actual position of the element (in whatever browser it was rendered) rather than the ‘requested’ value the CSS style contains.

Since the positions are relative to one another, it won’t do me much good to have the position in relation to the document. What i really need is the x/y positions of my parent element in relation to its own offset element. That is, I need position() value.

Notice one more thing: css(‘left’) will return the css value – a number and units – like ‘100px’, which is why I needed to encapsulate it with “parseFloat” before I continued with the calculations. On top of that, I run the risk of the css style returning ‘auto’ which would be parsed into 0 and render the entire mirroring action completely faulty.

position() and offset(), however, return x/y values, which are numerical, and represent the actual distances and positions relative to their parent and the screen.

VisualEditor Supercalifragilisti-Nested Divs

VisualEditor has lots of nested divs. And nested iFrames inside divs, that also have divs in them. And many of those get some position property along the chain; a Widget may get to be positioned at the center of the screen, while its panels are positioned with some percentage according to their size, and they, in turn, will have an input that has a popup attached.

And they’re all relative, and most of them has their positions injected dynamically, since the position most of the time depends on either where your marker is, or where your mouse is, or where whatever element you intend to edit is located, etc etc.

All of this makes flipping coordinates in VisualEditor challenging.

But the bigger challenge comes when we do that while calculating the offset and/or position() value of another element. We actually do that quite a lot, especially with popup widgets. For instance, inside the Page Settings window, you can add new Categories. Typing the category name in the input pops a “suggestion menu” that appears right under it. That means that for the suggestion menu to appear correctly, we need to check where the input element (which is it’s “sibling” in the nesting chain) is located, and mirror the coordinates. But we need to be careful and check that we don’t mirror mirror coordinates (because that won’t make sense) or mirror already mirrored mirror mirror coordina— well, you get my drift. It can be confusing.

On top of that, VisualEditor popups are iframes, and iframes are problematic with jQuery as it is, especially when they pop up already inside other iframes (re category popup inside the  Page Settings widget).

The more we depend on other dynamically positioned elements, the more we run the risk of having inconsistencies when mirroring those coordinates.

Example position() vs css()

Check out this StackOverflow answer. It shows an example of how position() and css() values are equal and another where they are be different.

The solution

Since my popup widget appeared inside another widget, the offset and general different in rendering was probably more emphasized than if the element appeared as part of the main VisualEditor surface. So, the solution was rather simple:

// ('dimensions' variable was set above according to ltr positions)
// Fix for RTL 
if ( this.$.css( 'direction' ) === 'rtl' ) {
	dimensions.right = this.$.parent().position().left - dimensions.width - dimensions.left;
	// Erase the value for 'left':
	delete dimensions.left;
}
this.$.css( dimensions );

And that worked like a charm.

Many thanks go to Timo Tijhof (Krinkle) who found this problem by spotting it in Chrome, and to Roan Kattouw (Catrope) for figuring out the solution is to use position().left 

More Resources

• Chrome vs Firefox, Position vs CSS (Stack Overflow) http://stackoverflow.com/questions/11860130/why-does-jquery-cssleft-and-position-left-return-different-values
  
• jQuery offset() http://api.jquery.com/offset/
• jQuery position() http://api.jquery.com/position/

The post Element Positioning: .css( ‘left’ ) vs .position().left (Or: Making Element Positioning Play Nice Across Browsers) appeared first on .

I’m excited to announce the first release of a product I’ve been working on for the last several months: the Miga Data Viewer. Miga (pronounced MEE-ga) is an open source application, mostly written in Javascript, that creates an automatic display interface around a set of structured data, where the data is contained in one or more CSV files. (CSV, which stands for “comma-separated values”, is a popular and extremely lightweight file format for storing structured data.) You supply the file(s), and a schema file that explains the type of each field contained in that file or files, and Miga does all the rest.

“Miga” means “crumb” in Spanish, and informally it can mean (I think) anything small and genial. The name helps to suggest the software’s lightweight aspect, though I confess that mostly I just picked the name because I liked the sound of it. (There’s also a resemblance to the name “MediaWiki”, but that is – I think – a coincidence.) As for the “data viewer” part, I could have called it a “data browser” instead of “data viewer” – in some ways, “browser” more accurately reflects what the software does – but that would have had the initials “DB”, which is already strongly associated with “database”. I also considered calling it a “data navigator” or “data explorer” instead, but I liked the compactness of “viewer”.

Conceptually, Miga is based almost entirely on the Semantic MediaWiki system, and on experiences gained working with it about how best to structure data. Miga began its life when I started thinking about what it would take to create a mobile interface for SMW data. It wasn’t long into that process when I realized that, if you’re making a separate set of software just for mobile display, that same set of software could in theory handle data from any system at all. That’s what led me to the “data is data” theory, which I wrote about here earlier this year. To summarize, it’s the idea that the best ways to browse and display a set of data can be determined by looking at the size and schema of the data, not by anything connected to its subject matter. And now Miga exists as, hopefully, a proof concept of the entire theory.

The practical implementation of Miga owes a lot to Semantic MediaWiki as well. The structure of the database, and the approach to data typing, are very similar to that of Semantic MediaWiki (though the set of data types is not identical – Miga has some types that SMW lacks, like “Start time” and “End time”, that make automatic handling easier). The handling of n-ary/compound data is based on how things are done in SMW – though in SMW the entities used to store such data are called either “subobjects” or “internal objects”, while in Miga they’re called part of “unnamed categories”. And the main interface is based in large part on that of the Semantic Drilldown extension.

You can think of Miga as Semantic MediaWiki without the wiki – the data entry is all required to have been done ahead of time, and the creation of logical browsing structures is done automatically by the software.

There’s another way in which Miga differs from SMW, though, which is that the data is stored on the browser itself, using Web SQL Database (a feature of browsers that really just means that you can store databases on the user’s own computer). The data gets stored in the browser when the site is first loaded, and from then on it’s just there, including if the user closes the browser and then navigates to the page again. It makes a huge difference to have the data all be stored client-side, instead of server-side: pages load up noticeably faster, and, especially on mobile devices, the lack of requirement of the network after the initial load has a big impact on both battery usage and offline usability – if you load the data in the browser on your cell phone, then head into an area with no coverage, you can keep browsing.

The website has more information on usage of the software; I would recommend looking at the Demos page to see the actual look-and-feel, across a variety of different data sets. I’ll include here just one screenshot

Hopefully this one image encapsulates what Miga DV can do. The software sees that this table of data (which comes from infobox text within the Wikipedia pages about public parks) contains geographical coordinates, so it automatically makes a mapping display available, and handles everything related to the display of the map. You can see above the map that there’s additional filtering available, and above that that one filter has already been selected. (And you can see here this exact page in action, if you want to try playing around with all the functionality.)

Miga DV is not the only framework for browsing through arbitrary structured data. Spreadsheets offer it to some extent, via “pivoting” and the like, including online spreadsheet applications like Google Docs. The application Recline.js offers something even closer, with the ability to do mapping, charting and the like, although the standard view is much closer to a spreadsheet than Miga’s is. There are libraries like Exhibit and Freebase Parallax that allow for browsing and visualization of data that’s overall more sophisticated than what Miga offers. Still, I think Miga offers an interface that’s the closest to the type of interface that people have become used to on the web and in apps, with a separate page for each entity. That, combined with the ease of setup for administrators, makes Miga a good choice in many situations, in my opinion.

There’s also the element that Miga is open-source software. I know less about what’s going among proprietary software in this field, but I wouldn’t be surprised if there’s similar software and/or services that costs money to use. There’s certainly no shortage of proprietary app-development software; the advantage of an open-source solution over paid software is a matter of personal opinion.

What next? There are a few features that I’m hoping to add soon-ish, the most important being internationalization (right now all the text displayed is hardcoded in English). In the longer term, my biggest goal for the software is the ability to create true mobile apps with it. There are a few important advantages that mobile apps have over web-based applications; the biggest, in my opinion, is they can be used fully offline, meaning even if the phone or device is shut off and then restarted somewhere with little or no reception. People do also like the convenience of having a separate icon for each app, though that can be replicated to some extent with URLs (which, as I understand, is easier to do on iOS than Android.)

My hope is of course that people start to make use of this software for their own data – both for public websites and for private installations, such as within corporations. Maybe Miga, or systems like it, will mean that a lot of data that otherwise would never be published, because creating an interface around it would take too much money and/or developer time, will finally get its day. And beyond that, it would be great if some sort of user and developer community grew around the software; but we’ll see.

I’ve spent a lot of my Physics education dealing with calculations. Calculating positions of objects as they move in space, particles in some electric field, planets around a certain star, and more, more and some more. In Physics, Mathematics is the language we use to represent reality, perform predictions, and validate them.

So it was only natural that when I deal with flips and mirroring when it comes to Right-to-Left vs. Left-to-Right support, I expected to handle these with ease. After all, after transforming relative frames and different types of physical field coordinates to one another, what’s transforming cartesian to mirror-cartesian?

Well, apparently it’s something.

Computer Coordinate System

Computer software, and especially web-based content, is originally meant to be read in English; left to right, top to bottom. When an application loads on any screen, it isn’t necessarily clear what the size of the visible content would be, especially with the advent of mobile devices of all shapes and sizes.

It makes sense, then, to start your coordinate system origin at the top-left corner, where the page begins to load, and go onwards from there — to the right, and to the bottom. So, going right on the axis increases X, and going down from the axis increases Y.

It makes perfect sense for computers to behave this way. Really. I get it. So.. what’s the problem, you ask?

The Problem

Eh. This difference between +y and -y makes using real-life calculation inside computer applications all that much more complicated. Whenever we deal with mathematics in physics and in the real world, we take our coordinate system as starting from the bottom-left corner. That means that going right on the origin increases the value of X, and going up from the origin increases the value of Y.

That’s exactly a horizontal mirror of what Computers expect.

This can be handled by flipping the value of y, of course ,but it is still the source of many a-frustrations, especially when programming applications that invlove physical formulas, the more elaborate the more complicated to keep track of. Also, this makes multiplications and additions — and the calculation of vector combinations – rather annoying.

But the issue becomes even more elaborate when we deal with complex positioning of elements that are nested in one another or are dynamically positioned. Not only do we need to deal with a flipped y, we also need to deal with dynamic calculations involving relative positioning.

Practical vs Conceptual

This problem of the flipped y axis is more an annoyance than an actual practical issue. Even if we deal with position calculations purely mathematically, there are ways to convert the Real World coordinates into computer-driven coordinates with some (quite basic) transformation function that mirrors the y’ values.

The main problem, though, is conceptual.

For example, when I work on improving RTL support in the new Visual Editor in MediaWiki, part of the issues I deal with are the locations of popups. The locations for these popups are often calculated dynamically, depending on where your marker is located.

You would think that this won’t cause any problem to an RTL content. After all, the x=10 position is still x=10 even in an RTL language. Whatever positions are calculated should remain in place even if the directionality of the page changed.

Not quite.

Relative Positioning

HTML elements are often relatively placed, especially when they’re nested inside other elements. For example, consider this HTML snippet:

<div id="master" style="position: absolute; width: 150px; height: 70px; left:0px;">
	Blah Blah Blah
   <div id="child" style=" position: relative; width: 50px; height: 50px; left: 150px;">Box</div>
</div>

The ‘child’ div is nested inside its ‘master’ div. The master is pushed all the way to the left, since it has “left=0” style value. The child has ‘left=150’, which would mean that the actual position of the child (assuming no other margins or any other parent positioning) is 150px from the left edge.

However, consider this:

<div id="master" style="position: absolute; width: 150px; height: 70px; right:0px;">
	Blah Blah Blah
   <div id="child" style=" position: relative; width: 50px; height: 50px; left: 150px;">Box</div>
</div>

Now the parent div is pushed all the way to the right of the screen, because of its “right:0” style. But the child div is still placed 150px from the left — this time, however, since it’s placed 150px to the left of its parent, it would be aaaaaaaaaaaall the way to the right +150px – essentially outside the screen.

Of course ,you can correct some of those problems with relative positioning, but relative positioning isn’t always what you need or want. And in the case of VisualEditor, many of the elements are surrounded by multiple other <div> elements that have their own calculated position values relative to the marker and the rest of the text. That’s on purpose, and it shoud stay this way.

Directionality Switching

So, why is the above example so important? Here’s why: When you switch content between LTR to RTL, you switch the directionality of the page. The ‘left’ style in the ‘master’ div above often has to be changed to ‘right’ so logical positions of text and buttons work out. Or, if you use something like CSSJanus, it may change automatically.

Example Bug: Visual Editor Link Popup in RTL

One of the bugs that I am trying to tackle in VisualEditor right now is the problem of the missing popup link in RTL wikis. If you edit an article and insert a link in LTR wiki (like English) a little popup will materialize, allowing you to change the target of the link and choose whether it is external or Wiki-Internal. Very cool and very useful.

In RTL, however, that link doesn’t pop up. It was fairly clear from the get-go that this is a CSS issue, some positioning problem that occurs when the page is “flipped” right-to-left instead of left-to-right — and, indeed, that is the case. While the parent’s position (defined in a CSS class) was flipped to be “right:0” in RTL, the popup itself still had a dynamically calculated “left” value. This conflict essentially put the entire popup at the outside of the screen.

The solution involved a simple change of popup direction style to the right, and recalculation of the distance from the new origin (right instead of left). Recalculating right to left when the marker is concerned is relatively easy. It looks something like this:

// Pseudocode:
if (langauge == 'rtl') {
   popup.right = window.innerWidth - popup.left;
   popup.left = '';
}

That solved the issue of the main popup, but it didn’t solve the issue of the subpopup. In VisualEditor, the link inspector is comprised of two popups: The main one involving the input in which you can enter your target page (inside the wiki or external http url) and a sub-popup that includes a list of suggestions.

Both popups have dynamically calculated values to the “left” in the LTR version. Both were changed to the ‘right’ with the above pseudocode. But what fixed the main popup, didn’t fix the sub-popup.

The sub popup appeared skewed, almost at the right place but not entirely, mocking me with its imperfect positioning.

Link popup and its suggestions sub-popup after RTL fix. It appears — but not quite right

The Solution

It took me a while (and some gentle coaxing from my mentors) to realize what was happening, and when I did, it seemed rather obvious.

The main popup calculates its position in relation to the marker, and the sub-popup calculates its position in relation to the first popup — both popups are inside a parent div that is, also, dynamically placed.

The mirroring should not be calculated in relation to the entire screen, because the positions (left or right) are set in relation to whatever container div holds the popups and not in relation to the entire screen.

The only reason I didn’t notice this fact with the main popup is that its skew’yness wasn’t all that evident, it appeared a little to the right but still inside the selected word. But when the second popup is misaligned, it’s noticeable.

The mirroring should be done in relation to whatever container holds the popup, and not assume that the position is mirrored across the entire screen.

The new code, then, is something like this:

// Pseudocode:
if (langauge == 'rtl') {
   popup.right = parentContainer.innerWidth - popup.left;
   popup.left = '';
}

And here’s the visual result:

Much better!

The RTL/LTR Vertigo

Regardless of the actual solution, this problem raised an important issue: LTR to RTL switches can involve some dynamic calculations that may be somewhat messy, and when we switch the directionality of an entire page, some elements in it may flip (those who have CSS classes, mostly) and some may not (those who have dynamically calculated positions that are injected inline).

That means that examining the code and trying to calculate the transformation of the mirroring can be a truly confusing experience, one that I hereby name “The LTR/RTL Vertigo“.

Some elements flip, some don’t, some values require change, some don’t, some are relative to their parents and some relative to a grand-parent or the content box itself. And then, of course, there’s the fact that the x/y axis are the mirror of what I am used to from my Physics background. Did I say Vertigo already?

But don’t worry, this condition is mostly harmless, mostly temporary, and is mostly solved with a quick break from the screen and exposure to the summer sun to remind oneself that there is, in fact, a world out there.

So, until next time – Beware the RTL/LTR Vertigo, think of relative positioning, calculate your math mirrored, and go outside a little. It’s actually kinda nice.

The post The Issues of Coordinate Systems (or: Mirroring XY, Oh My!) appeared first on .

This year I had the distinct privilege of being accepted to the Google Summer of Code summer internship, working for the Wikimedia Foundation. I will be responsible for improving Right-to-Left support in the new Visual Editor.

It’s really fun work, though quite challenging, and the fact my fixes (when and if they pass review) are implemented in the system and are used by Wikipedians all over the world, is quite satisfying.

If you’re interested, you can see the full accepted proposal and my MediaWiki user page, and get all the bi-weekly technical updates as I go in the updates page.

I’ll try to update as I go, too, so if you want to hear more, stick around and stay tuned!

The post Google Summer of Code 2013: RTL Support for MediaWiki’s Visual Editor appeared first on .

My new motto for 2013 is: “data is data”. What does that mean (aside from being a tautology)? It means that data has a set of behaviors and a “personality” all it own, very much distinct from its underlying subject matter, or from what format it’s in. Ultimately, the fact that a table of data holds an ID field, three string fields and a date, and that it has 300 rows, says more about how to display and interface with it than the fact that it’s an elementary school activity listing, or a description of video game characters, or top-secret military information, or the results of biotech research. And the fact that the data is in an Excel spreadsheet, or a database, or embedded as RDF in a web page, shouldn’t matter either.

What is data?

I should first define what I mean by data. I’m talking about anything that is stored as fields – slots where the meaning of some value can be determined by the name or location of that value. And it should be information that represents something about the outside world.

For that latter reason, I don’t think that information related to communication – whether it’s email, blog posts, microblogs and status updates, discussion forums and so on – is truly data, though it’s usually stored in databases. It’s not intended to represent anything beyond itself – there’s data about it (who wrote it and when, etc.), but the communication itself is not really data. Tied in with that, there’s rarely a desire to display such communications in a venue other than where it was originally created. (Communication represents the vast majority of information in social networking sites, but it’s not the only information – there’s also real data, like users’ friends, interests, biographical information and so on.)

Data can be stored in a lot of different ways. I put together a table of the different terms used in different data-storage approaches:

What’s the most obvious observation here (other than maybe the fact that this too is a table of data)? That, for all of their differences, all of these storage mechanisms are dealing with the same things – they just have different ways to refer to them.

A wrapper around a database

The vast majority of websites that have ever been created have been, at heart, code around a database, where the code is mostly intended to display and modify the contents of that database. That’s true of websites from Facebook to eBay to Craigslist to Wikipedia. There’s often an email component as well, and sometimes a credit card handling component, and often peripherals like ads, but the basic concept is: there’s a database somewhere, and users can use the site to navigate around some or all of its contents, some or all users can also use the site to add or modify contents. The data structure is fixed (though of course it changes, usually getting more complex, over time), and often, all the code to run the site had to be created more or less from scratch.

Of course, not all web products are standalone websites: there’s software to let you create your own blogs, wikis, e-commerce sites, social networking sites, and so on. This software is more generic than standalone websites, but it, too, is tied to a very specific data structure.

So you have millions of hours, or possibly even billions, that have been spent creating interfaces around databases. And in a lot of those projects, the same sort of logic has been implemented over and over again, in dozens of different programming languages and hundreds of different coding styles. This is not to say that all of that work has been wasted: there has been a tremendous amount of innovation, hard work and genius that has gone into all of it, optimizing speed, user interface, interoperability and all of that. But there has also been a lot of duplicated work.

Now, as I noted before, not all data stored in a database should be considered data: blog posts, messages and the like should not, in my opinion. So my point about duplicated work in data-handling may not full apply to blogs, social networking sites and so on. I’m sure there’s needlessly duplicated work on that side of things as well, but it’s not relevant to this essay. (Though social-networking sites like Facebook do include true structured data as well, about users’ friends, interests, biographical information, etc.)

What about [insert software library here]?

Creating a site, or web software, “from scratch” can mean different things. There are software libraries that work with a database schema, making use of the set of tables and their fields to let you create code around a database without having to do all the drudgework of creating a class from scratch from every table, etc. Ruby on Rails is the most well-known example, but there’s a wide variety of libraries in various languages that do this sort of thing: they are libraries that implement what’s known as the active record pattern. These “active record” libraries are quite helpful when you’re a programmer creating a site (I myself have created a number of websites with Ruby on Rails), but still, these are tools for programmers. A programmer still has to write code to do anything but the most basic display and editing of information.

So here’s a crazy thought: why does someone need to write any code at all, to just display the contents of a database in a user-friendly manner? Can’t there be software that takes a common-sense approach to data, displaying things in a way that makes sense for the current set of data?

No database? No problem

And, for that matter, why does the underlying data have to be in a database, as nearly all web software currently expects it to be? Why can’t code that interfaces with a database work just as well with data that’s in a spreadsheet, or an XML file, or available through an API from some other website? After all, data is data – once the data exists, you should be able to display it, and modify it if you have the appropriate permissions, no matter what format it’s in.

It’s too slow to query tens of thousands of rows of data if they’re in a spreadsheet? Fine – so have the application generate its own database tables to store all that data, and it can then query on that. There’s nothing that’s really technically challenging about doing that, even if the amount of data stretches to the hundreds of thousands of rows. And if the data or data structure is going to change in the outside spreadsheet/XML/etc., you can set up a process to have the application keep re-importing the current contents into its internal database and delete the old stuff, say once a day or once a week.

Conversely, if you’re sure that the underlying data isn’t being modified, you could have the application also allow users to modify its data, and then propagate the changes back to the underlying source, if it has permissions to do so.

Figuring out the data structure, and other complications

Now, you may argue that it’s not really possible to take, say, a set of Excel spreadsheets and construct an interface out of it. There’s a lot we don’t know: if there are two different tables that contain a column called “ID”, and each one has a row with the value “1234″ for that column, do those rows refer to the the same thing? And if there’s a column that mostly contains numbers, except for a few rows where it contains a string of letters, should that be treated as a number field, or as a string? And so on.

These are valid points – and someone who wants to use a generic tool to display a set of data will probably first have to specify some things about the data structure: which fields/columns correspond to which other fields/columns, what the data type is for each field, which fields represent a unique ID, and so on. (Though some of that information may be specified already, if the source is a database.) The administrator could potentially specially all of that “meta-data” in a settings file, or via a web interface, or some such. It’s some amount of work, yes – but it’s fairly trivial, certainly compared to programming.

Another complication is read-access. Many sites contain information that only a small percentage of its users can access. And corporate sites of course can contain a lot of sensitive information, readable only to a small group of managers. Can all of that read-access control really be handled by a generic application?

Yes and no. If the site has some truly strange or even just detailed rules on who can view what information, then there’s probably no easy way to have a generic application mimic all of them. But if the rules are basic – like that a certain set of users cannot view the contents of certain columns, or cannot view an entire table, or cannot view the rows in a table that match certain criteria, then it seems like that, too, could be handled via some basic settings.

Some best practices

Now, let’s look at some possible “best practices” for displaying data. Here are some fairly basic ones:

• If a table of data contains a field storing geographical coordinates (or two fields – one for latitude and one for longitude), chances are good that you’ll want to display some or all of those coordinates in a map.
• If a table of data contains a date field, there’s a reasonable chance that you’ll want to display those rows in a calendar.
• For any table of data holding public information, there’s a good chance that you’ll want to provide users with a faceted search interface (where there’s a text input for each field), or a faceted browsing/drill-down interface (where there are clickable/selectable values for each field), or both, or some combination of the two.

If we can make all of these assumptions, surely the software can too, and provide a default display for all of this kind of information. Perhaps having a map should be the default behavior, that happens unless you specify otherwise?

But there’s more that an application can assume than just the need for certain kinds of visualization interfaces. You can assume a good amount based on the nature of the data:

• If there are 5 rows of data in a table, and it’s not a helper table, then it’s probably enough to just have a page for each row and be done with it. If there are 5,000 rows, on the other hand, it probably makes sense to have a complete faceted browsing interface, as well as a general text search input.
• If there are 10 columns, then, assuming you have a page showing all the information for any one row of data, you can just display all the values on that one page, in a vertical list. But if you have 100 columns, including information from auxiliary tables, then it probably makes sense to break up the page, using tabs or “children” pages or just creative formatting (small fonts, use of alternating colors, etc.)
• If a map has over, say, 200 points, then it should probably be displayed as a “heat map”, or a “cluster map”, or maps should only show up after the user has already done some filtering.
• If the date field in question has a range of values spread out over a few days, then just showing a list of items for each day makes sense. If it’s spread out over a few years, then a monthly calendar interface makes sense. And if it’s spread out over centuries, then a timeline makes sense.

Somehow software never makes these kinds of assumptions.

I am guilty of that myself, by the way. My MediaWiki extension Semantic Drilldown lets you define a drill-down interface for a table of data, just by specifying a filter/facet for every column (or, in Semantic MediaWiki’s parlance, property) of data that you want filterable. So far, so good. But Semantic Drilldown doesn’t look at the data to try to figure out the most reasonable display. If a property/column has 500 different values, then a user who goes to the drilldown page (at Special:BrowseData) will see 500 different values for that filter that they can click on. (And yes, that has happened.) That’s an interface failure: either (a) those values should get aggregated into a much smaller number of values; or (b) there should be a cutoff, so that any value that appears in less than, say, three pages should just get grouped into “Other”; or (c) there should just be a text input there (ideally, with autocompletion), instead of a set of links, so that users can just enter the text they’re looking for; or… something. Showing a gigantic list of values does not seem like the ideal approach.

Similarly, for properties that are numbers, Semantic Drilldown lets you define a set of ranges for users to click on: it could be something like 0-49, 50-199, 200-499 and so on. But even if this set of ranges is well-calibrated when the wiki is first set up, it could become unbalanced as more data gets added – for example, a lot of new data could be added, that all has a value for that property in the 0-49 range. So why not have the software itself set the appropriate ranges, based on the set of data?

And maybe the number ranges should themselves shift, as the user selects values for other filters? That’s rarely done in interfaces right now, but maybe there’s an argument to be made for doing it that way. At the very least, having intelligent software that is aware of the data it’s handling opens up those kinds of dynamic possibilities for the interface.

Mobile and the rest

Another factor that should get considered (and is also more important than the underlying subject matter) is the type of display. So far I’ve described everything in terms of standard websites, but you may want to display the data on a cell phone (via either an app or a customized web display), or on a tablet, or on a giant touch-screen kiosk, or even in a printed document. Each type of display should ideally have its own handling. For someone creating a website from scratch, that sort of thing can be a major headache – especially the mobile-friendly interface – but a generic data application could provide a reasonable default behavior for each display type.

By the way, I haven’t mentioned desktop software yet, but everything that I wrote before, about software serving as a wrapper around a database, is true of a lot of enterprise desktop software as well – especially the kind meant to hold a specific type of data: software for managing hospitals, amusement parks, car dealerships, etc. So it’s quite possible that an approach like this could be useful for creating desktop software.

Current solutions

Is there software (web, desktop or otherwise) that already does this? At the moment, I don’t know of anything that even comes close. There’s software that lets you define a data structure, either in whole or in part, and create an interface apparatus around it of form fields, drill-down, and other data visualizations. I actually think the software that’s the furthest along in that respect is the Semantic MediaWiki family of MediaWiki extensions, which provide enormous amounts of functionality around an arbitrary data structure. There’s the previously-mentioned Semantic Drilldown, as well as functionality that provides editing forms, maps, calendars, charts etc. around an arbitrary set of data. There are other applications that do some similar things – like other wiki software, and like Drupal, which lets you create custom data-entry forms, and even like Microsoft Access – but I think they all currently fall short of what SMW provides, in terms of both out-of-the-box functionality and ease of use for non-programmers. I could be wrong about that – if there’s some software I’m not aware of that does all of that, please let me know.

Anyway, even if Semantic MediaWiki is anywhere near the state of the art, it still is not a complete solution. There are areas where it could be smarter about displaying the data, as I noted before, and it has no special handling for mobile devices; but much more importantly than either of those, it doesn’t provide a good solution for data that doesn’t already live in the wiki. Perhaps all the world’s data should be stored in Semantic MediaWiki (who am I to argue otherwise?), but that will never be the case.

Now, SMW actually does provide a way to handle outside data, via the External Data extension – you can bring in data from a variety of other sources, store it in the same way as local data, and then query/visualize/etc. all of this disparate data together. I even know of some cases where all, or nearly all, of an SMW-based wiki’s data comes from externally – the wiki is used only to store its own copy of the data, which it can then display with all of its out-of-the-box functionality like maps, calendars, bulleted lists, etc.

But that, of course, is a hack – an entire wiki apparatus around a set of data that users can’t edit – and the fact that this hack is in use just indicates the lack of other options currently available. There is no software that says, “give me your data – in any standard format – and I will construct a pleasant display interface around it”. Why not? It should be doable. Data is data, and if we can make reasonable assumptions based on its size and nature, then we can come up with a reasonable way to display it, without requiring a programmer for it.

Bringing in multiple sources

And like SMW and its use of the External Data extension, there’s no reason that the data all has to come from one place. Why can’t one table come from a spreadsheet, and another from a database? Or why can’t the data come from two different databases? If the application can just use its own internal database for the data that it needs, there’s no limit to how many sources it was originally stored in.

And that also goes for public APIs, that provide general information that can enrich the local information one has. There are a large and growing number of general-information APIs, and the biggest one by far is yet to come: Wikidata, which will hold a queriable store of millions of facts. How many database-centered applications could benefit from additional information like the population of a city, the genre of a movie, the flag of a country (for display purposes) and so on? Probably a fair number. And a truly data-neutral application could display all such information seamlessly to the user – so there wouldn’t be any way of knowing that some information originally came from Wikidata as opposed to having been entered by hand by that site’s own creators or users.

Data is data. It shouldn’t be too hard for software to understand that, and it would be nice if it did.

If you use Semantic MediaWiki, or are curious about it, I highly recommend going to SMWCon, the twice-yearly conference about Semantic MediaWiki. The next one will be a month and half from now, in New York City – the conference page is here. I will be there, as will Jeroen De Dauw, who will be representing both core SMW developers and the extremely important Wikidata project; as will a host of SMW users from corporations, the US government, startups and academia. There will be a lot of interesting talks, the entrance fee is quite reasonable ($155 for a three-day event), and I’m the local chair, so I can tell you for sure that there will be some great evening events planned. (And the main conference will be at the hacker mecca ITP, which is itself a cool spot to check out if you’ve never been there.) I hope some of you can make it!

I am happy to announce a project I’ve been working on for a rather long time now: Working with MediaWiki, a general-purpose guide to MediaWiki. It finally was released officially two days ago. It’s available in print-on-demand (where it numbers roughly 300 pages), e-book (.epub and .mobi formats) and PDF form.

As anyone who knows WikiWorks and our interests might expect, Semantic MediaWiki and its related extensions get a heavy focus in the book: a little less than one third of the book is about the Semantic MediaWiki-based extensions. I think that’s generally a good ratio: anyone who wants to learn about Semantic MediaWiki can get a solid understanding of it, both conceptually and in practice; while people who don’t plan to use it still get a lot of content about just about every aspect of MediaWiki.

This book is, in a sense, an extension of our consulting business, because there’s a lot of information and advice contained there that draws directly on my and others’ experience setting up and improving MediaWiki installations for out clients. There’s a section about enabling multiple languages on the same wiki, for instance, which is a topic I’ve come to know fairly well because that’s a rather common request among clients. The same goes for controlling read- and write-access. Conversely, there is only a little information devoted to extensions that enable chat rooms within wikis, even though there are a fair number of them, because clients have never asked about installing chat stuff.

So having this book is like having access to our consultants, although of course at a lower price and with all the benefits of the written word. (And plenty of illustrations.) And I think it’s a good investment even for organizations that do work with us, to get the standard stuff out of the way so that, when it comes time to do consulting, we can focus on the challenging and unique stuff.

Once again, here is the book site: Working with MediaWiki. I do hope that everyone who’s interested in MediaWiki checks it out – my hope is that it could make using MediaWiki simpler for a lot of people.

Today is the launch date of a new WikiWorks site: Innovention wiki, which “showcases the themes of innovation and invention through stories drawn from South Australia.”

Skinning

We were able to design a really nice skin for the site, based on the specs of their designer. It uses a 3-column layout which is kind of uncharted territory as far as MediaWiki skins go. Part of the challenge here was the right-hand column. The search section is a part of the skin, while the maps, photos and videos are generated by the MediaWiki page itself. This was accomplished by putting that stuff into a div which uses absolute positioning.

Another challenge was trying to fit a decent form into a very narrow middle column. The solution was to hide the right column via CSS, since the search form doesn’t really need to be on a form page. Then, the middle column is stretched to cover both columns. This was easy to do, since Semantic Forms helpfully adds a class to the body tag for any formedit page (which works for existing pages) and MediaWiki adds a tag to any Special page (for when adding a new place with Special:FormEdit). So the content area was accessed with:

.action-formedit #content, .mw-special-FormEdit #content {
   width: (a whole lot);
}

Displaying different stuff to logged in and anonymous users

While on the topic of body attributes, MediaWiki does not add any classes to the body tag which would differentiate logged in from anonymous users. This doesn’t present a problem for the skin, which can easily check if the user is logged in. But what if you wanted to have a part of the MediaWiki content page displayed only for anonymous users? A common example would be exhortations to create an account and/or sign in. That’s something that should be hidden for logged in users. Fortunately, this is easily and cleanly resolved.

Since this was a custom skin, we overrode the Skin class’s handy addToBodyAttributes function (hat tip):

function addToBodyAttributes( $out, $sk, &$bodyAttrs ) {
$bodyClasses = array();

/* Extend the body element by a class that tells whether the user is
logged in or not */
if ( $sk->getUser()->isLoggedin() ) {
   $bodyClasses[] = 'isloggedin';
} else {
   $bodyClasses[] = 'notloggedin';
}

if ( isset( $bodyAttrs['class'] ) && strlen( $bodyAttrs['class'] ) > 0 ) {
   $bodyAttrs['class'] .= ' ' . implode( ' ', $bodyClasses );
} else {
   $bodyAttrs['class'] = implode( ' ', $bodyClasses );
}

return true;
}

For the built in skins, this is still easy to do. Just use the same code with the OutputPageBodyAttributes hook in your LocalSettings.php. This function adds a class to the body tag called either “isloggedin” or “notloggedin.” Then add the following CSS to your MediaWiki:SkinName.css:

.isloggedin .hideifloggedin {
   display:none
}
.notloggedin .hideifnotloggedin {
   display:none
}

Now in your MediaWiki code simply use these two classes to hide information from anonymous or logged in users. For example:

<span class="hideifnotloggedin">You're logged in, mate!</span>
<span class="hideifloggedin">Dude, you should really make an account.</span>

Combine with some nifty login and formedit links

Or even better, here’s a trick to generate links to edit the current page with a form:

<span class="hideifnotloggedin"> [{{fullurl:{{FULLPAGENAMEE}}|action=formedit}} Edit this page]</span>

…and a bonus trick that will log in an anonymous user and THEN bring him to the form edit page:

<span class="hideifloggedin">[{{fullurl:Special:Userlogin|returnto={{FULLPAGENAMEE}}&returntoquery=action=formedit}} Log in and edit this page.]</span>

It doesn’t get much better than that! See it in action here. Yes, you’d have to make an account to really see it work. So take my word for it.

Spam bots

While on the subject of making an account, it seems that bots have gotten way too sophisticated. One of our clients had been using ConfirmEdit with reCAPTCHA and was getting absolutely clobbered by spam. I’ve found that for low traffic wikis, the best and easiest solution is to combine with QuestyCaptcha instead. They’re easily broken by an attacker who is specifically targeting that wiki, but very few wikis have gained that level of prominence. The trick is to ask a question that only a human can answer. I’ve had success with this type of question:

Please write the word, “horsse”, here (leave out the extra “s”): ______

Featured article slideshow

This site has a pretty cool main page. The main contributor to that coolness is the transitioning slideshow with various featured articles. Gone are the days when a wiki only featured one page! This was made possible by bringing the Javascript Slideshow extension up to date, which was done by our founder Yaron Koren in honor of Innovention wiki. The articles are inserted manually which gives the user complete control over the appearance. But it would be pretty simple to generate the featured pages with a Semantic MediaWiki query.

Tag Cloud

Also on the main page is a nifty tag cloud. That is done with Semantic Result Formats and its tag-cloud format (by our own Jeroen De Dauw). Maybe I’ll blog more about that as it develops.

Stay tuned

The site will be developed further over the next few weeks, with some neat stuff to come…

I regret to say that our consultant Jeroen De Dauw will not be doing any significant work for WikiWorks for at least the next year. Thankfully, that’s for a very good reason: he’s moved to Berlin to be part of the Wikidata project, which starts tomorrow.

Wikidata is headed by Denny Vrandecic, who, like Jeroen, is a friend and colleague of mine; and its goal is to bring true data to Wikipedia, in part via Semantic MediaWiki. There was a press release about it on Friday that got some significant media attention, including this good summary at TechCrunch.

I’m very excited about the project, as a MediaWiki and SMW developer, as a data enthusiast, and simply as a Wikipedia user. This project quite different from any of the work that I’ve personally been involved with, because Wikipedia is simply a different beast from any standard wiki. There are five challenges that are specific to Wikipedia: it’s massive, it needs to be extremely fast at all times, it’s highly multi-lingual (over 200 languages currently), it requires references for all facts (at least in theory), and it has, at this point, no real top-down structure.

So the approach they will take will be not to tag information within articles themselves, the way it’s done in Semantic MediaWiki, but rather to create a new, separate site: a “Data Commons”, where potentially hundreds of millions of facts (or more?) will be stored, each fact with its own reference. Then, each individual language Wikipedia can make use of those facts within its own infobox template, where that Wikipedia’s community sees fit to use it.

It’s a bold vision, and there will be a lot of work necessary to pull it off, but I have a lot of faith in the abilities of the programmers who are on the team now. Just as importantly, I see the planned outcome of Wikidata as an inevitable one for Wikipedia. Wikipedia has been incrementally evolving from a random collection of articles to a true database since the beginning, and I think this is a natural step along that process.

A set of files were discovered in 2010 that represented the state of Wikipedia after about six weeks of existence, in February 2001. If you look through those pages, you can see nearly total chaos: there’s not even a hint of a unifying structure, or guidelines as to what should constitute a Wikipedia page; over 10% the articles related in some way to the book Atlas Shrugged, presumably added by a devoted fan.

11 years later, there’s structure everywhere: infobox templates dictate the important summary information for any one subject type, reference templates specify how references should be structured, article-tagging templates let users state precisely the areas they think need improvement. There are guidelines for the first sentence, for the introductory paragraphs (ideally, one to four of them, depending on the article’s overall length), for how detailed sections should be, for when one should link to years, and so on. There are also tens of thousands of categories (at least, on the English-language Wikipedia), with guidelines on how to use them, creating a large set of hierarchies for browsing through all the information. These are all, in my eyes, symptoms of a natural progression toward a database-like system. Why is it natural? Because, if a rule makes sense for one article, it probably makes sense for all of them. Of course, that’s not always true, and there can be sub-rules, exceptions, etc.; but still, there’s no use reinventing the wheel for every article.

People complain that the proliferation of rules and guidelines, not to mention categories and templates, drive away new users, who are increasingly afraid to edit articles for fear of doing the wrong thing. And they’re right. But the solution to this problem is not to scale back all these rules, but rather to make the software more aware of the rules, and the overall structure, to prevent users from being able to make a mistake in the first place. That, at heart, was the thinking behind my extension Semantic Forms: if there’s a specific way of creating calls to a specific template, there’s no point requiring each user to create them in that way, when you can just display a form, let the user only enter valid inputs, and have the software take care of the rest.

Now, Wikidata isn’t concerned with the structuring of articles, but only with the data that they contain; but the core philosophy is the same: let the software take care of anything that there’s only one right way to do. If a country has a certain population (at least, according to some source), then there’s no reason that the users of every different language Wikipedia need to independently look up and maintain that information. If every page about a mutiplanetary system already has its information stored semantically, then there’s no reason to separately maintain a hand-generated list of multiplanetary systems. And if, for every page about a musician, there’s already semantic information about their genre, instrument and nationality, then there’s no reason for categories such as “Danish jazz trumpeters“. (And there’s probably much less of a need for categories in general.)

With increased meaning/semantics on one hand, and increased structure on the other, Wikipedia will become more like a database that can be queried, than like a standard encyclopedia. And at that point, the possibilities are endless, as they say. The demand is already there; all that’s missing is the software, and that’s what they’ll be working on in Berlin. Viel Glück!

This is something that comes up, particularly when dealing with MediaWiki infoboxes. We had an infobox table floating on the right side of the page with a fixed with. Then there was an image to the left of it that was supposed to take up the remaining page’s width. The challenge was: What happens as the user shrinks or stretches the browser window? The fixed width table would stay the same size but the image would have to grow or shrink with the browser width.

There are some scripts out there that would do this. You don’t need them. Here’s what to do:

{| style="float: right; width: 613px" ... |}

<div id="image-holder" style="margin-right: 630px;">
[[File:Image.jpg]]
</div>

Then add some CSS:

div#image-holder img {
  height: auto;
  width: 100%;
}

Simple, right? Basically, the 100% width combined with the right margin gives us a “100% percent minus x number of pixels” effect. The browser responds accordingly.

See it live here.