Plan:

• Brief introduction to each framework
• Followed by discussion focused on use cases
• Maurits will live blog (thanks!)
• Framework vs. Product - let’s not worry about that!

For each system:

• What use cases is it well suited to
• What use cases is it poorly suited to

Discussion:

• Focus on client use cases and what frameworks would be a good or poor fit
• You can also ask questions!

Live blogging.

Use cases:

• Non-profit with custom forms and import/export
• Concert venue with strict event listing and advanced settings.
• NGO needs a database application for their highly confidential data.
• Generate a map out of data, and store extra information, like images for the data.
• Three-person company with basically static pages, non-technical users.
• Real-time application with websockets.
• A large company wants to monitor how well their customer service staff are doing. Strict form with statistics on it.

Pyramid:

• Best for APIs, not really CMS or websites, but applications
• Start small, grow
• Complex non-profit use case: you would have to build everything yourself.
• Event listing: we have a site that manages stuff for music bands as example, you just need some endpoints. Maybe websauna on top.
• NGO privacy: security really good, and simple enough to understand the full system. You can fully build your own fine grained security system. UI is a lot of work.
• 100 person company wants intranet with various content, installable by local IT team.
• Intranet: try to create a good solution for multiple clients, not tailored to one in a consulting project. Pyramid is good for building such products.
• Map: pyramid does not care which database you use. In a project we are mapping electical grids. In Guillotina they think they are nice with AsyncIO, but it makes their whole codebase harder to understand.
• Real-time: we had websockets for a year, but it was too slow, so we rewrote it in Go-Lang.
• Form statistics: you can create a form, store it in a database, export it, fine.

Django:

• Aimed at 80% of use cases, simple auth system
• Good for CRUD apps, smooth beginning
• Non-profit: export/import is good, we can do forms. Sharing access gets tricky
• Event listing: Django can do it, but just use Wordpress. But the technical user can do this in the admin interface.
• NGO privacy: permissions are too simple OOTB, not per object. You may want end-to-end encryption, which no framework offers. Limited built-in audit log, nice start.
• Intranet: not good fit for Plone for the people who do not often work with it. Django has easier CMSes for this. The expenses claim form could be easier to integrate into Django. Deployment just needs Postgres. DjangoCMS is Plone Light. Wagtail is WordPress Plus.
• Map: any of the frameworks except Plone can do the geo stuff in postgres. Images just on the file system. Static and uploaded files are handled fine by Django, can also be in the cloud.
• Real-time: not many Django users will care about websockets, will never be the focus.
• Form statistics: you can but I would not do it. This feels like a custom web application. Do it in any framework.

Guillotina:

• Small framework, scale from small to big
• Non-profit with forms: decent fit, use JSON schema
• Event listing: go to wix.com or something, it seems too simple
• NGO privacy: Good fit, permissions are no problem for this. You need to build the full UI. Okay, you may be able to start with Volto as UI, but that needs integration in the backend.
• Intranet: backend can handle it, need to build UI, deployment can be with docker and kubernetes.
• Map: for the large files, Guillotina wins because of AsyncIO.
• Real-time: we use websockets just fine.
• Form statistics: you can build an app, depending on your UI skills.

Plone:

• Good when you have different use groups with complex requirements for security.
• Edit interface is doable, but less so on mobile.
• Import/export complicated, but you can hire Jens.
• Difficult to find skilled people.
• Non-profit with forms: really good fit, end-users can create the forms. The import/export would need help from a provider.
• Event listing: do not bother with Plone. It sounds more relational, Plone would be overkill.
• NGO privacy: you need deep understanding of Plone security, hierarchy for storing data. Audit logging with add-on.
• Intranet: Plone is fine. You may need to help with installing at first, but it will keep running once installed.
• Small almost static site: none of us.
• Real-time: once Asko is done with his ZServer improvements, we can do it.
• Form statistics: do not use Plone. Use sed, awk, perl. Technically you could use some add-ons.

Three PAS plugins:

• collective.denyroles
• pas.plugins.headers
• collective.contentgroups

What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.

It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.

There are about twenty different plugin types, and they interact with each other.

In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active

So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.

Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)

We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.

We wanted collective.workspace to manage roles through groups instead of sharing.

And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.

It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.

So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.

So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.

And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.

There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.

Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.

Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.

To keep the data structure up to date, we use events.

To recap:

• remove external connections when possible
• know the limitations of your plugins
• you can patch PAS to avoid some plugins
• you can use a custom group plugin

Further ideas:

• lazily create collective.workspace groups
• lazily fetch group and user properties and roles

Takeaways:

• PAS is great, but mixing and abusing plugins can be deadly
• Sometimes replace plugins with a new one that does exactly what you need.

At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.

Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.

Clients:

• Large university site
• High-profile government client
• Large research institution client website

Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.

We use Jenkins to test the migration.

General backend things:

• Old ATTopics to core Collections. Use collective.jsonify to export it, in a way that already looks more like a Collection.
• Rich text: migrate to Volto blocks.
• Post migration: collective.cover needed special handling.

What we polish to enter in Volto land:

• We use collective folderish types.
• deal with default pages
• Convert RichText to Volto DraftJS
• easily point to old website when content is not imported
• fix some urls (we want to use resolveuid)
• simple folders we should turn info page with listing block
• simple collections we should turn info page with collection block

Question: can you open source the module that handles the RichText to Volto blocks migration?

Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.

See the annual report.

Jen Myers has graciously offered to stay on next year as treasurer.

The Zope Foundation has been added to the Plone Foundation. Still ongoing, complex process.

With the Pylons community, those talks have stalled at the moment. Might happen in the future.

Financially we lost more money than we usually do. There was a trademark conflict that cost a lot more than we thought.

Some money should come from the Zope Foundation by the way.

Results of the vote for the board.

48 valid votes, one invalid. 2 went to the spam folder, but we found them. 30 percent of votes were in paper.

We had a vote by the foundation membership. Voted into the Plone board have been: Victor, Erico, Chrissy, Andy, Paul, Jens, Fulvio.

Meeting closed by Paul.

Thanks to Alexander, Kim and Carol who are leaving the board.

We wanted to find an easy way to reproduce the installation process. We use:

• Vagrant
• Fabric
• AWS
• Boto 3 / awscli

Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.

Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.

More libraries with Fabric:

• fabtools to make sure that for example Python is installed, or a user is created.
• Cuisine: update files. You can also use this tool to ensure packages and users, so parts are very similar to fabtools.

Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.

I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.

I have some upgrade code that you can use, or for some parts copy and adapt for your use case.

Code: https://github.com/collective/collective.migrationhelpers

Upgrade steps:

• Do imports in the function.
• Do them conditional, so it does not fail if some package does not exist.
• Do not let a step fail when run a second time.

You may want to disable LDAP temporarily in an upgrade step.

Get the birds eye view, with some code that reports:

• how many items are there
• which portal types
• how big
• local roles
• etcetera
• How many items are there that need to be replaced or removed, like PloneFormGen or Collage.

Divide and conquer:

• Deal with one problem at a time.
• Ignore problems that don't block you. You may try to solve something in the old Plone 4.3 site which is already fixed just by completing the migration to Plone 5.

You can register upgrade steps conditionally in zcml if needed, for example with zcml:condition="installed plone-52".

Make big problems small:

• Write something that removes 98 percent of your content, for testing. Keep the structure in place though: Folders may have portlets or local roles that give problems that you want to know of.
• Do not migrate blobs. The PDF that lives on the filesystem will not change. Move the blobstorage out of the way, use experimental.gracefulblobmissing during migration, and move blobstorage in again.
• Copy the Data.fs.index too, if it is a really big site.

Forget the past:

• remove all revisions
• maybe manually use collective.revisionmanager for this.
• pack the database with zero days (bin/zeopack keeps the last day by default).
• Remove no longer needed portlets.
• You can use uninstall and upgrade profiles. I like having the upgrade code in Python, but sometimes a profile is much easier.
• Remove utilities and adapters from add-ons that will be removed.
• Sometimes you cannot easily remove a package. You can make an alias for it. plone.app.upgrade has code for this, for example to not crash on an old site that still expects the kupu editor package somewhere.

Migrating LinguaPlone:

• Content editors may have done crazy things, combining folders and content from different languages.
• We have code to migrate this to plone.app.multilingual in migration.plonehelpers.

Update to Plone 5.2:

• Use Python 2.7 at first.
• Include Archetypes if needed for migration.
• Run the migration to migrate to dexterity.

Archetypes to dexterity:

• Use the methods from pac_migration from plone.app.contenttypes.
• Start with the containers/folders.
• Do one type at a time.
• Especially for blob-like items there are options to speed this up, like disabling updating SearcheableText.
• You can write custom migrators, which might just need a few lines.

Alternative: inplace migrator from ftw.upgrade. Interesting for large folders. Might be a nice PR to get this part into core.

To Python 3:

• We need to remove some archetypes tools.
• Then remove the Archetypes eggs from buildout and use Python 3.
• Run the zodbupdate script.
• When using RelStorage, you may want to switch to filestorage temporarily.
• Read the documentation please, especially: - upgrade to Python 3 - upgrade zodb to Python 3

Why do it this way? I don't want every Plone company to have their own migration code stashed away somewhere. Please contribute to the core.

Upgrade to Plone 6? Install plone.restapi, use Volto, done.

A year ago, ZServer was one of the last Zope parts that did not run on Python 3. The idea was to use WSGI instead. I wondered why Python 3 would not be possible. So I tried porting it. It worked. And I added websocket support. You can use this to automatically push pages from Plone to Gatsby. Or show a notification when a page needs review, via content rules.

I replaced the old Medusa server with Twisted. ZServer was optionally using Twisted years ago, but this was ripped out, and I don't know why. But I got it working again. So for HTTP and webdav we have a Twisted http server and thread pool with an async FileSender, an AsyncIO/uvloop when avilable, standard ZConfig configuration and logging, And HTTP/2 works as well.

Bonus: websockets with PubSub messaging. It uses the Autobahn Twisted WebSocket protocol. ZeroMQ PubSub cockets using IPC-socket. The connection autosubscribes to local events. The server publishes a guarded event, for example indicating that an event is restricted to Editors.

One problem at this moment with the wsgi setup using waitress, is that the requests are limited to the number of wsgi threads, so server large files can become problematic. There are probably ways around this, but ZServer does not have this problem.

Status:

• My ZServer fork is at https://github.com/datakurre/ZServer. See mostly branch datakurre/master.
• You will need a special branch of plone.recipe.zope2instance as well.
• Plus collective.wsevents, plonectl, collective.taskqueue.

Upstream:

• In Zope 4, WebDav was made dependent on ZServer, by mistake?
• Zope 5 assumes ZServer and WebDav no longer exist.

Remaining steps:

• Get it upstream or not?
• Eliminate dead code.
• QA: tests, documentation
• Release it.
• Restore wsgi=off on Python 3.

I would be interested to hear when others are interested in using this. It is better if more people use this.

I am an introvert from Finland, so I did not ask the Zope people, or visited the Zope sprints.

Jens: There was no real reason to drop ZServer, except that no one has tried it because the code was so very old. Now you got it working, let's get this into Zope.

See the slides.

It is hard to see what part of machine learning is over hyped, and what part is actually useful.

Basis of ML (Machine Learning) is to get input and transform it to output. The ML gets a picture of a duck, and it gives as answer "duck". That is image recognition. You train the ML with images that you already label. And then you give it an image, and hope it gives a good answer. Same: from English to Japanese text. From an fMRI scan to an image visualized.

Simpler example: solar panel. Input: time of day, output: how much generated power. You would train this with data. The ML would turn this into a function that gives an approximately good answer.

Simplest model: linear regression. Turn the data into a function like this:

a * X + b

No line through the training points will be perfect. You find the line that minimizes the average error. So the ML uses linear regression to find a and b.

In our example, we try to guess the amount of mojitos we sell, based on the number of passengers that a daily boat brings to our beach bar.

But this may also depend on the temperature, or the number of sharks. With two variables you would not have a line, but a plane. With n, you would get an n-dimensional shape. You have n inputs, and give a weight to each input, and add them.

This is about numbers. For image recognition, you don't get a number as output. But we can apply a function to the result, and get a number between 0 and 1 that gives a likelyhood. For an image we may have 0.02 certainty that it is a cat, and 0.9 that it is a duck, so our answer will be the highest: a duck. Translated to the mojitos: what is the likelyhood that my bar breaks even?

This system with weights and a reducer function is called a perceptron. With a short Python program I got 90 percent accuracy on the standard NDIST test. We can do better.

We look at neural networks. This is basically: mash two perceptrons together. Or more. We are finding a function that approximates the data, and reduce the error iteratively.

What would I say is deep learning? Why has it grown?

1. Neural networks with many layers...
2. ... trained with a lot of data...
3. ... with specialized architectures.

We helped Facebook with neural networks by tagging our photos. We gave them training data!

A lot of engineering is going into deep learning.

Generative Adversarial Networks: GANs. Example: a horse discriminator. Train the system with images of horses and others, and it should answer with: yes or no horse.

Other part: horse generator. Randomly generate images for feeding to the horse discriminator. You train this by saying: you did or did not manage to trick the discriminator. And you try to get better. I trained this a night, and after about a million iterations, I got pictures that are not quite horses, there is something wrong, but they are amazingly close.

Is Volto compatible with Guillotina? Not 100 percent. Question is how we keep the APIs in sync. If people figure out a shared generic API that works with both Plone and Guillotina as backend, this can work.

How do you migrate? You can migrate from Plone 4.3 to Volto. We did that with transmogrifier. Biggest problem is moving composite pages, like cover, to the Volto blocks. In a post migration step, you can fix things up.

Why Semantic UI? Rob researched all the popular ones, concentrating on how easy it is to theme or override stuff. Semantic UI makes this easy. You derive from a theme, and only override some parts.

For add-ons, how do you keep the backend part in sync with the frontent part? You don't. You use Python packages in the backend and node packages in the frontend. There will probably be a lot of packages that are only in the backend or only in the frontend. You may need to be careful in the frontend so that it can work with several versions of the backend of this add-on.

TypeScript has won the JavaScript wars. Will you support that? We don't want to support two ways of doing the same thing. So if we switch to TypeScript, we want to stop using ES6, also in the documentation. Same for class based versus function based.

Did you check accessibility? Yes. We have a static code checker ALM that helped us fix issues. Also the Cyprus tool. But we also do manual checks. Plone Foundation is trying to get funding for an audit.

Gatsby Preview gives you hot reloading of content. So you change something in Plone CMS, and it immediately is visible on your Gatsby site.

Plone CMS -> Web hook -> Gatsby source plugin. When content is created, edited, deleted, we will fire a websocket event.

When you change the title of a folder, you should update the breadcrumbs of the children too, and also navigation in several places. It took me a long time, but I have implemented this.

I have created a Gatsby theme for Plone: https://github.com/collective/gatsby-theme-plone

Frontend is nice, but it needs a backend, so let's talk about that. I will talk about plonecli, plone.api, Plone snippets for VS Code and plone.restapi.

You should use plonecli. It saves a lot of time for boring stuff. It helps you to create a product and enhance it step by step. It can cleanup and create a fresh virtualenv with a specific Python version and requirements and run buildout.

It has sub templates, for example to add a content type to your package. Now a sub template for a restapi service. Also one for upgrade steps, adding an upgrade profile per upgrade step.

We create a structure and files, and if you don't use some parts you can ignore or remove them. We will check if you have a clean git status before, and ask to commit, and afterwards we automatically commit.

You have good test coverage right from the start. All features added by plonecli have at least basic test coverage. You only write the tests for your own code. We make a tox environment to test different Plone and Python versions.

You can configure plonecli/mr.bob to your taste, changing the default answers or ignoring questions in a .mrbob file.

It is extensible, you can write your own custom bobtemplate packages and register them for the plonecli.

Visual Studio Code: snippets for Plone

There are snippets for schemas, fields, registry xml.

Use plone.api. It makes add-on code much easier to understand, without arcane incantations.

Ideas for the future.

• plonecli:
  • add option to set Interface for view, viewlet.
  • REST API sub templates for de/serializer.
  • Graphical UI (ncurses-like) to make selecting options easier.
• VS Code: plone.api autocompletion. VS Code is not so smart with namespaces. And it does not know buildout. You can use a buildout recipe that helps here though. Or use the generated zopepy script as your Python.

I would love more contributions. Don't be shy. There is no grand jury who makes decisions. Publish your own bobtemplates. Improve the VS Code snippets, or for other editors. Meet me at the sprints.

Question: is there a bobtemplate to create a new bobtemplate?

Answer: No. Should be possible.

https://ugent.be is a Plone 4.3 site of a large Belgian university. Started in 2002 as Zope/CMF site. 90.000 pages, sub sites, 40.000 students, hundreds of editors, 90 add-ons. They wanted to move to Plone 5.2 and Python 3.

• Traditional in-place migration: too manu add-ons, no one-on-one mapping possible.
• Transmogrifier: not yet Python 3 at the time, too much magic hidden in too many places with blueprints.
• So: custom migration solution.

Content types: standard types, plus four custom content types, including PloneFormGen. So that is quite reasonable. There is extensive usage of archetypes schema extenders.

Start: analyze and investigate your dependencies. - Based on Archetypes? Obsolete, replace. - No longer needed? Remove it. In Confluence we compiled a big table with for each package the basic information of how we would handle it: upgrade it, replace it, unknown yet, status of Python support.

Start with a minimal Plone 5.2 setup. Add one verified Python 3 compatible add-on at a time. Test extensively. Focus on content types first. Things like portlets can be handled later.

You need an export. We used a customized version of collective.jsonify. Core numbers: 90.000 json files, 55 GB data, 90 minutes, binary files base64 encoded.

We exported portlet assignments, default pages, layout information, workflow state, local roles, and pre-computed values for further efficient processing.

So we had 90.000 json files on the file system. We imported this in ArangoDB. Why use such a migration database? This allowed us to import only some portal types, or do parallel imports, and test complex migration steps like for PloneFormGen.

We briefly tried MongoDB, but that could not handle data over 16 MB. The json could be dumped unchanged in ArangoDB. This took 45 minutes.

Now we need to import this into Plone. Clean Python 3.7, Plone 5.2, plus the minimal set of packages needed for the content types. Import via plone.restapi. On top we have a dedicated migration package with special views. This handled things like translating between UIDs and paths.

The "magic" migration script is based on configuration in YAML.

• Phase 1: pre-check the migration, remove target site if it already exists from previous test, create new Plone Site, install add-ons.
• Phase 2: create all Folders, query ArangoDB for this.
• Phase 3: create all non folders.
• Phase 4: global actions. Check and migrate paths to UIDs in rich text fields. Assign portlets. Other specific fixup operations, like reindexing.

We migrated PloneFormGen (Archetypes) to easyform (dexterity). Export: one JSON for the FormFolder, one JSON file per field and action adapter. In easyform this needed to be turned into an EasyForm instance and a schema.

Topics (AT) to Collections (dexterity). Code was largely taken over from the plone.app.contenttypes migration.

From AT schema extenders to dexterity behaviors. First we made a list of which there were, are they in use, what do they do? Check which dexterity replacements there are. Create new behaviors.

Migrate packages to Python 3. This is mostly covered by talks of Philipp Bauer and David Glick. Common problems: utf-8 versus unicode, import fixes, implements to implementer. I rarely used the 2to3 and modernizr tools.

Some reimplementations: - portal skins to browser views - some packages with AT replaced with new packages with dexterity

Other problems:

• improper file and image metadata
• migration of vocabulary values, like old to new departments
• repetitive cycles: always a bug occurs after a day of migration right before the end.

Quality control: - you need to check that migrated content and configuration is complete - "works for me" is nice, but others need to check too

Most of the packages have been removed, the setup is much smaller.

Status: - content migration is complete - must be tested in detail - integrate with the new theme, test this - need a replacement of a specific membrane usage - need work on castle/cas plugin

Takeaways: - Export Plone to JSOJN: 2 hours. Fast. - Import JSON to ArangoDB: 45 minutes. Fast. - Import ArangoDB to plone.restapi: 36-48 hours. Painfully slow. - 1.5 - 2.0 seconds per content object on average - cannot parellellize this import, because you would get conflict errors - So Plone and ZODB and painfully slow for creating mass content.

Question: in a similar setup we did a live migration from a live site to a separate new site. Did you consider that?

Answer: I tried this for other sites, but here I wanted to be able to partial imports, independent of the live site.

Question: default migration patches away all kinds of expensive indexing. You might want to consider looking at that. Migrating ten items per second is possible, although that is inline migration. And can you share the code, especially for the easyform migration?

Answer: could be done, but is not the primary focus of the budget currently.

A slightly older version of the code is on community.plone.org.