Earlier this month, I released an update of my PrintCSS Portal print-css.rocks.
Thiis update includes support for new tools like
I also want to mention the portal printcss.live by Andreas Zettl. printcss.live is a very nice and supplementary addition to the PrintCSS community because it allows you to test tools directly within the browser. Andreas Zettl also maintains a list a PrintCSS articles on medium.com.
Ghost Logo
After almost 12 years I moved my personal blog blog.zopyx.com from Plone 4 to the Ghost blogging platform. You may ask why? First, my old Plone 4 installation reached end of lifetime a long time ago and it was time for moving the content to a different system. I investigated a lot of time into options like static site generators like Eleventy, Huge and others but I was not very much happy with them. I just wanted a simple system where I can add and publish content easily.
The migration from Plone 4 to Ghost was a bit painful. First, I tried to use plone.restapi or collective.jsonify but the Plone 4 site gave me a bunch of headaches and errors. So I decided to create a static copy of the original bog using httrack. With some Python code I was able to extract the content part from the HTML markup of each document and some necessary metadata like publication date, title, description etc. All extracted data was written to a JSON file.
Ghost itself provides some JSON format as import/export format. Due to limited resources I hired a Ghost developer (for a very small budget) that converted my JSON format into the Ghost JSON format.
Then setting up Ghost (running in a Docker container) behind Apache was more or less straight-forward. As with every new system: you need to learn and try things out.
For the moment, I am happy with the result and outcome. There are clearly some issues like a better theme or some CSS fixes to be implemented.
I hope to blog again more in the future.
This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org. The conference will be held from 5th to 13th December 2020. No matter where you are, you can participate.
Some preliminary information (might change still):
The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences too
Stay tuned for more information! Follow @ploneconf and #ploneconf2020 on Twitter, Instagram, and TikTok!
Zope is the original open source web application server written in Python, featuring a component architecture and an object-oriented, hierarchical data model.
This release
Although this release is a major one, it should be possible to use it without any changes if the following preconditions are met:
Your project runs on the latest Zope 4 release on Python 3.6+ and does not use ZServer.
Neither your tests nor running your Zope instance shows DeprecationWarnings.
Additionally, this release contains some minor features and bug fixes that have been backported to Zope 4 where possible.
As Zope 5 is now released, Zope 4 is put into bugfix mode until 2021-12-31.
(See the roadmap for details: https://zope.readthedocs.io/en/latest/roadmap.html 3)
For details about this release see the what’s-new-section and the changelog:
To install the new version see https://zope.readthedocs.io/en/latest/INSTALL.html
Plone 5.2.2 is a bug fix release of Plone 5.2. Release Manager for this version is Maurits van Rees.
Note: We want to do releases more often, to avoid having very big changes between two bugfix releases. Plone 5.2.3 is expected in October 2020. We announce pending releases for testing on https://community.plone.org, and we announce a final release there once it is out of the pending phase, even when the installers are not ready yet.
Installers are being made, so not all links below will work yet. Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.2/versions.cfg.
For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/
This release packs a lot of changes: the previous release was from January 2020. Some highlights are:
For detailed changelog, go to https://plone.org/download/releases/5.2.2
Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.
This initiative is described in more detail in the Board‘s statement on the Plone governance process.
The first Steering Circle meeting was held on Zoom on 22nd September 2020. Representatives from each community team were invited and 22 people from around the world attended. Discussion items included:
Volto is Plone's snappy, modern React front end, and the Volto team has added two major new features to the latest release.
Beginning with Plone 4, site admins have been able to create customized content types easily in the browser using Plone's standard administrative UI. Now, this feature has been added to Volto. You can:
In short, it's possible to customize a content type in every possible way, and the resulting type is automatically searchable and addable.
Blocks are Volto's way of adding and organizing various elements on a page. They can be text elements, images, videos, dynamic lists, video embeds, or whatever you decide to develop. In Volto 8, you can combine the features of blocks editing with your newly created content type, vastly enhancing the functionality and customizability of your custom content - all in the browser with just a couple of clicks.
Download Volto from Github or kick the tires on the Volto demo site.
As some Plone community members know, one of our own, Matthew Wilkes, has been hard at work writing a book for the last year. We're very excited to announce that it has been published! It is Advanced Python Development: Using Powerful Language Features in Real-World Applications and it can be purchased at Matthew's website, Apress or Amazon.
The book follows the design and build of a real-world application example from prototype to production quality. In this context, it explains Python language features that aren’t routinely covered and introduces techniques that demonstrate large scale system design and development processes. Plus there are useful asides, best practices and library recommendations galore.
Matthew specializes in Python development, especially in the areas of web security and performance. He has been developing Python web applications for 15 years and has been very active in the open-source community in general and the Plone community in particular.
Matthew began using Plone in 2004 when it was at version 2.0. He has worked on various aspects of the core Plone software and was on the Plone framework and security teams through the Plone 4 era. He led the security team for over four years, modernizing security team practices by instituting unit testing of fixes across all versions, cooperation with Red Hat and other downstream vendors, and a pre-announcement policy. He has also contributed to the security of Zope, Django CMS and Pyramid, including recently rewriting Pyramid’s CSRF protection. This all gives him a wider view of security as a process rather than just deep knowledge of one system.
Matthew served as the administrator for Plone's participation in Google Summer of Code for many years. He has also served on the Plone Foundation's Membership Committee and Board of Directors, and he was instrumental in helping to overcome the many legal and logistical hurdles to bringing Zope under the Plone Foundation's umbrella.
Matthew is a welcome and frequent presence at Plone conferences and sprints and has been known to be involved in adventures in late night drinking. For example after the party at the first Bristol conference, Matthew was instrumental in arranging for the pub where many Plonistas were hanging out to continue serving them after official closing time. The party continued until 5 in the morning and we learned later that the place took in £16,000 that night. Matthew also collaborated with David Glick on the classic zodb.ws April Fool’s Day launch in 2011 when they managed to get a hacked-up version of the ZODB running in the browser - a prank which required both technical chops and a particular sense of humor.
Congratulations to Matthew!
This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org.
Some preliminary information (might change still):
The conference will be held from 14th to 22nd November 2020. No matter where you are, you can participate.
Stay tuned for more information soon!
Call for papers is open, please submit your talk proposal at
https://2020.ploneconf.org
And one more thing...
Stay tuned for more information!
The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.
A ZODB can contain objects that cannot be loaded. Reasons for that may be:
The most frequent issues are caused by:
You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.
Real problems may be revealed at that point, e.g when:
You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:
In short these approaches to fixing exist:
I will mostly focus on the last approach.
But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.
Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!
In the simplest form run it like this:
$ bin/zodbverify -f var/filestorage/Data.fs
It will return:
Note
zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:
$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs
$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs
The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:
$ ./bin/zodbverify -f var/filestorage/Data.fs
[...]
INFO:zodbverify:Done! Scanned 163955 records.
Found 1886 records that could not be loaded.
Exceptions, how often they happened and which oids are affected:
ModuleNotFoundError: No module named 'Products.Archetypes': 1487
0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]
ModuleNotFoundError: No module named 'Products.PloneFormGen': 289
0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]
AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34
0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f
ModuleNotFoundError: No module named 'Products.CMFDefault': 20
0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131
ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20
0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10
ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16
0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f
ModuleNotFoundError: No module named 'fourdigits': 8
0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316
ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3
AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3
0x2a712b 0x2a712c 0x2a712d
ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1
0x11
ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1
0x25
ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1
0x37
ModuleNotFoundError: No module named 'plone.app.controlpanel': 1
0x0f4c2b
ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1
0x3b1311
You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.
You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.
Before the summary the log dumps a huge list of errors that contain the pickle and the error:
INFO:zodbverify:
Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):
INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\[email protected]]q6X\x05\x00\x00\x00Ownerq7asu.'
INFO:zodbverify:Traceback (most recent call last):
File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record
class_info = unpickler.load()
File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class
return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'Products.PloneFormGen'
In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?
If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.
But usually it is not simply content but something else. Here are some examples:
The hardest part is to find out what and where the broken object actually is before removing or fixing it.
The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.
A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.
A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.
With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.
More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.
And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.
To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.
Before we look at the path of 0x376b77 we'll inspect the object.
Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:
$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D
INFO:zodbverify:Inspecting 0x376b77:
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
INFO:zodbverify:
Object as dict:
{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'[email protected]': ['Owner']}}}
INFO:zodbverify:
The object is 'obj'
[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()
-> pickle, state = storage.load(oid)
(Pdb++)
Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.
You can now inspect it:
(Pdb++) obj
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
(Pdb++) pp obj.__dict__
{'__Broken_newargs__': (),
'__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',
'__ac_local_roles__': {'[email protected]': ['Owner']},
'_at_creation_flag': True,
'_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',
'_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,
'_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',
'at_references': <Folder at at_references>,
'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),
'demarshall_hook': None,
'id': 'thank-you',
'includeEmpties': True,
'marshall_hook': None,
'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),
'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,
'portal_type': 'FormThanksPage',
'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,
'showAll': True,
'showFields': [],
'showinsearch': True,
'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,
'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,
'title': 'Danke'}}
If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.
Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.
Continue to let zodbverify find the path to the object:
INFO:zodbverify:Building a reference-tree of ZODB...
[...]
INFO:zodbverify:Created a reference-dict for 163955 objects.
INFO:zodbverify:
This oid is referenced by:
INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1
INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2
INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3
INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4
INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5
INFO:zodbverify:0x01 OFS.Application.Application at level 6
INFO:zodbverify: 8< --------------- >8 Stop at root objects
INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7
INFO:zodbverify: 8< --------------- >8 Stop at root objects
INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8
INFO:zodbverify: 8< --------------- >8 Stop at root objects
INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6
INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7
INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8
INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9
INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10
INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9
INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3
INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4
INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5
INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6
INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7
INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8
INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9
INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2
INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3
INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4
INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5
INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6
INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7
INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8
INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9
INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!
You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.
This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".
In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.
Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).
The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.
I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.
At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.
Use that when a module or class has moved or was renamed.
Docs: https://github.com/zopefoundation/zodbupdate
You can change objects in DB according to rules:
Examples from Zope/src/OFS/__init__.py:
zodbupdate_decode_dict = {
'OFS.Image File data': 'binary',
'OFS.Image Image data': 'binary',
'OFS.Application Application title': 'utf-8',
'OFS.DTMLDocument DTMLDocument title': 'utf-8',
'OFS.DTMLMethod DTMLMethod title': 'utf-8',
'OFS.DTMLMethod DTMLMethod raw': 'utf-8',
'OFS.Folder Folder title': 'utf-8',
'OFS.Image File title': 'utf-8',
'OFS.Image Image title': 'utf-8',
'OFS.Image Pdata title': 'utf-8',
'OFS.Image Pdata data': 'binary',
'OFS.OrderedFolder OrderedFolder title': 'utf-8',
'OFS.userfolder UserFolder title': 'utf-8',
}
zodbupdate_rename_dict = {
'webdav.LockItem LockItem': 'OFS.LockItem LockItem',
}
You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.
Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7
Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39
You can inject a module to work around missing or moved classes or modules.
The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.
Examples in __init__.py:
# -*- coding: utf-8 -*-
from OFS.SimpleItem import SimpleItem
from plone.app.upgrade.utils import alias_module
from plone.app.upgrade import bbb
from zope.interface import Interface
class IBBB(Interface):
pass
class BBB(object):
pass
SlideshowDescriptor = SimpleItem
# Interfaces
try:
from collective.z3cform.widgets.interfaces import ILayer
except ImportError:
alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)
try:
from App.interfaces import IPersistentExtra
except ImportError:
alias_module('App.interfaces.IPersistentExtra', IDummy)
try:
from webdav.interfaces import IDAVResource
except ImportError:
alias_module('webdav.interfaces.IDAVResource', IDummy)
# SimpleItem
try:
from collective.easyslideshow.descriptors import SlideshowDescriptor
except ImportError:
alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)
# object
try:
from collective.solr import interfaces
except ImportError:
alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)
try:
from Products.CMFPlone import UndoTool
except ImportError:
sys.modules['Products.CMFPlone.UndoTool'] = bbb
More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py
Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)
If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.
from persistent import Persistent
from ZODB.utils import p64
import transaction
app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]
for oid in broken_oids:
dummy = Persistent()
dummy._p_oid = p64(oid)
dummy._p_jar = app._p_jar
app._p_jar._register(dummy)
app._p_jar._added[dummy._p_oid] = dummy
transaction.commit()
You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.
from persistent import Persistent
from ZODB.utils import p64
import transaction
app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]
for oid in broken_oids:
root = connection.root()
del app._p_jar[p64(oid)]
transaction.commit()
I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.
This is how you should deal with most problems.
-o <OID> -D to inspect one object and find out where that object is referencedreferenced by until you find where in the tree the object lives. zodbverify will try to do it for you.The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8
When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.
The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.
If 0x3b1d06 is the broken oid inspect it with zodbverify:
$ ./bin/instance zodbverify -o 0x3b1d06 -D
2020-08-24 12:19:32,441 INFO [Zope:45][MainThread] Ready to handle requests
2020-08-24 12:19:32,442 INFO [zodbverify:222][MainThread]
The object is 'obj'
The Zope instance is 'app'
[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()
-> pickle, state = storage.load(oid)
(Pdb++) obj
<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>
(Pdb++) pp [i for i in obj]
[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,
[...]
<class 'webdav.interfaces.IDAVResource'>,
<InterfaceClass plone.dexterity.interfaces.IDexterityContent>,
<InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,
[...]
<SchemaClass plone.supermodel.model.Schema>]
The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.
When you press c for continue zodbverify will proceed and load the pickle:
(Pdb++) c
2020-08-24 12:20:50,784 INFO [zodbverify:68][MainThread]
Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):
2020-08-24 12:20:50,784 INFO [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'
2020-08-24 12:20:50,786 INFO [zodbverify:70][MainThread] Traceback (most recent call last):
File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record
unpickler.load()
File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class
return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package
0: \x80 PROTO 3
2: ( MARK
3: c GLOBAL 'OFS.EtagSupport EtagBaseInterface'
38: q BINPUT 1
40: c GLOBAL 'Acquisition.interfaces IAcquirer'
74: q BINPUT 2
76: c GLOBAL 'plone.app.dexterity.behaviors.discussion IAllowDiscussion'
135: q BINPUT 3
137: c GLOBAL 'zope.annotation.interfaces IAnnotatable'
178: q BINPUT 4
180: c GLOBAL 'zope.annotation.interfaces IAttributeAnnotatable'
230: q BINPUT 5
232: c GLOBAL 'plone.uuid.interfaces IAttributeUUID'
270: q BINPUT 6
272: c GLOBAL 'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'
327: q BINPUT 7
329: c GLOBAL 'Products.CMFCore.interfaces ICatalogAware'
372: q BINPUT 8
374: c GLOBAL 'Products.CMFCore.interfaces ICatalogableDublinCore'
426: q BINPUT 9
428: c GLOBAL 'zope.location.interfaces IContained'
465: q BINPUT 10
467: c GLOBAL 'Products.CMFCore.interfaces IContentish'
508: q BINPUT 11
510: c GLOBAL 'OFS.interfaces ICopySource'
538: q BINPUT 12
540: c GLOBAL 'webdav.interfaces IDAVResource'
572: q BINPUT 13
574: c GLOBAL 'plone.dexterity.interfaces IDexterityContent'
620: q BINPUT 14
622: c GLOBAL 'plone.app.relationfield.interfaces IDexterityHasRelations'
681: q BINPUT 15
683: c GLOBAL 'plone.dexterity.interfaces IDexterityItem'
726: q BINPUT 16
728: c GLOBAL 'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'
791: q BINPUT 17
793: c GLOBAL 'plone.dexterity.interfaces IDexteritySchema'
838: q BINPUT 18
840: c GLOBAL 'plone.app.contenttypes.interfaces IDocument'
885: q BINPUT 19
887: c GLOBAL 'Products.CMFCore.interfaces IDublinCore'
928: q BINPUT 20
930: c GLOBAL 'Products.CMFCore.interfaces IDynamicType'
972: q BINPUT 21
974: c GLOBAL 'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'
1040: q BINPUT 22
1042: c GLOBAL 'z3c.relationfield.interfaces IHasIncomingRelations'
1094: q BINPUT 23
1096: c GLOBAL 'z3c.relationfield.interfaces IHasOutgoingRelations'
1148: q BINPUT 24
1150: c GLOBAL 'z3c.relationfield.interfaces IHasRelations'
1194: q BINPUT 25
1196: c GLOBAL 'plone.namedfile.interfaces IImageScaleTraversable'
1247: q BINPUT 26
1249: c GLOBAL 'OFS.interfaces IItem'
1271: q BINPUT 27
1273: c GLOBAL 'plone.app.iterate.interfaces IIterateAware'
1317: q BINPUT 28
1319: c GLOBAL 'plone.portlets.interfaces ILocalPortletAssignable'
1370: q BINPUT 29
1372: c GLOBAL 'zope.location.interfaces ILocation'
1408: q BINPUT 30
1410: c GLOBAL 'OFS.interfaces IManageable'
1438: q BINPUT 31
1440: c GLOBAL 'Products.CMFCore.interfaces IMinimalDublinCore'
1488: q BINPUT 32
1490: c GLOBAL 'Products.CMFCore.interfaces IMutableDublinCore'
1538: q BINPUT 33
1540: c GLOBAL 'Products.CMFCore.interfaces IMutableMinimalDublinCore'
1595: q BINPUT 34
1597: c GLOBAL 'plone.app.content.interfaces INameFromTitle'
1642: q BINPUT 35
1644: c GLOBAL 'App.interfaces INavigation'
1672: q BINPUT 36
1674: c GLOBAL 'Products.CMFCore.interfaces IOpaqueItemManager'
1722: q BINPUT 37
1724: c GLOBAL 'AccessControl.interfaces IOwned'
1757: q BINPUT 38
1759: c GLOBAL 'AccessControl.interfaces IPermissionMappingSupport'
1811: q BINPUT 39
1813: c GLOBAL 'persistent.interfaces IPersistent'
1848: q BINPUT 40
1850: c GLOBAL 'OFS.interfaces IPropertyManager'
1883: q BINPUT 41
1885: c GLOBAL 'plone.app.relationfield.behavior IRelatedItems'
1933: q BINPUT 42
1935: c GLOBAL 'AccessControl.interfaces IRoleManager'
1974: q BINPUT 43
1976: c GLOBAL 'plone.contentrules.engine.interfaces IRuleAssignable'
2030: q BINPUT 44
2032: c GLOBAL 'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'
2097: q BINPUT 45
2099: c GLOBAL 'OFS.interfaces ISimpleItem'
2127: q BINPUT 46
2129: c GLOBAL 'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'
2196: q BINPUT 47
2198: c GLOBAL 'OFS.interfaces ITraversable'
2227: q BINPUT 48
2229: c GLOBAL 'plone.uuid.interfaces IUUIDAware'
2263: q BINPUT 49
2265: c GLOBAL 'Products.CMFEditions.interfaces IVersioned'
2309: q BINPUT 50
2311: c GLOBAL 'plone.app.versioningbehavior.behaviors IVersioningSupport'
2370: q BINPUT 51
2372: c GLOBAL 'Products.CMFCore.interfaces IWorkflowAware'
2416: q BINPUT 52
2418: c GLOBAL 'OFS.interfaces IWriteLock'
2445: q BINPUT 53
2447: c GLOBAL 'OFS.interfaces IZopeObject'
2475: q BINPUT 54
2477: c GLOBAL 'zope.interface Interface'
2503: q BINPUT 55
2505: c GLOBAL 'plone.dexterity.schema.generated Plone_0_Document'
2556: q BINPUT 56
2558: c GLOBAL 'plone.supermodel.model Schema'
2589: q BINPUT 57
2591: t TUPLE (MARK at 2)
2592: q BINPUT 58
2594: \x85 TUPLE1
2595: q BINPUT 59
2597: \x85 TUPLE1
2598: q BINPUT 60
2600: \x85 TUPLE1
2601: q BINPUT 61
2603: . STOP
highest protocol among opcodes = 2
If you are into this you can read the pickle now :)
If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:
(Pdb++) c
2020-08-24 12:22:42,596 INFO [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06
2020-08-24 12:22:42,597 INFO [zodbverify:43][MainThread] Building a reference-tree of ZODB...
2020-08-24 12:22:42,964 INFO [zodbverify:60][MainThread] Objects: 10000
2020-08-24 12:22:44,167 INFO [zodbverify:60][MainThread] Objects: 20000
2020-08-24 12:22:44,521 INFO [zodbverify:60][MainThread] Objects: 30000
2020-08-24 12:22:44,891 INFO [zodbverify:60][MainThread] Objects: 40000
2020-08-24 12:22:45,184 INFO [zodbverify:60][MainThread] Objects: 50000
2020-08-24 12:22:45,507 INFO [zodbverify:60][MainThread] Objects: 60000
2020-08-24 12:22:45,876 INFO [zodbverify:60][MainThread] Objects: 70000
2020-08-24 12:22:46,403 INFO [zodbverify:60][MainThread] Objects: 80000
2020-08-24 12:22:46,800 INFO [zodbverify:60][MainThread] Objects: 90000
2020-08-24 12:22:47,107 INFO [zodbverify:60][MainThread] Objects: 100000
2020-08-24 12:22:47,440 INFO [zodbverify:60][MainThread] Objects: 110000
2020-08-24 12:22:47,747 INFO [zodbverify:60][MainThread] Objects: 120000
2020-08-24 12:22:48,052 INFO [zodbverify:60][MainThread] Objects: 130000
2020-08-24 12:22:48,375 INFO [zodbverify:60][MainThread] Objects: 140000
2020-08-24 12:22:48,665 INFO [zodbverify:60][MainThread] Objects: 150000
2020-08-24 12:22:48,923 INFO [zodbverify:60][MainThread] Objects: 160000
2020-08-24 12:22:49,037 INFO [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.
2020-08-24 12:22:49,386 INFO [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json
2020-08-24 12:22:49,424 INFO [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:
0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1
0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2
0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3
0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8
8< --------------- >8 Stop at root objects
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8
0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9
0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10
0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10
0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11
0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10
0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11
0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3
[...]
From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.
fsoids.pyIn this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.
$ ./bin/zodbverify -f var/filestorage/Data.fs
INFO:zodbverify:Done! Scanned 120797 records.
Found 116 records that could not be loaded.
Exceptions and how often they happened:
AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20
AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3
ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
ModuleNotFoundError: No module named 'Products.Archetypes': 5
ModuleNotFoundError: No module named 'Products.CMFDefault': 20
ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1
ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1
ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1
ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1
ModuleNotFoundError: No module named 'fourdigits': 8
ModuleNotFoundError: No module named 'plone.app.controlpanel': 2
ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34
ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16
Follow the white rabbit...
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d
oid 0x35907d BTrees.OIBTree.OISet 1 revision
tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043
tid user=b'Plone [email protected]'
tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'
new revision BTrees.OIBTree.OISet at 282469
tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
tid user=b''
tid description=b''
referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038
[...]
Follow referenced by ...
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b
[...]
referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790
[...]
referenced by 0x11c284 BTrees.OOBTree.OOBTree
[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284
[...]
referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket
[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6
[...]
referenced by 0x11c278 z3c.relationfield.index.RelationCatalog
[...]
Found it!!!!!
fsoids.pyIn this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)
Then find out where exists in the tree by following the trail of items that reference it:
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00
oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision
tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
tid user=b''
tid description=b''
new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984
referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339
references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984
references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984
references 0x372e5b OFS.Folder.Folder at 912841984
references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984
references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984
references 0x372e5e Persistence.mapping.PersistentMapping at 912841984
tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976
tid user=b'Plone pbauer'
tid description=b'/Plone/rename_file_ids'
referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012
Follow referenced by until you find something...
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51
oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision
[...]
Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184
oid 0x280184 five.intid.intid.IntIds 1 revision
tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
tid user=b''
tid description=b''
new revision five.intid.intid.IntIds at 8579054
references 0x28018c <unknown> at 8579054
references 0x28018d <unknown> at 8579054
tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824
tid user=b' adminstarzel'
tid description=b'/Plone/portal_quickinstaller/installProducts'
referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690
referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834
That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.
Here is a example of how to remove all broken objects from the catalog in a pdb-session:
(Pdb++) from zope.intid.interfaces import IIntIds
(Pdb++) from zope.component import getUtility
(Pdb++) intid = getUtility(IIntIds)
(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]
(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)
(Pdb++)
(Pdb++) import transaction
(Pdb++) transaction.commit()
After packing the DB the problem is gone. o/
zodbbrowser to inspect the ZODB.The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.
There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.
The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.
from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
def remove_relations(context=None):
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()
For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py
Many addons and features in Plone store data in Annotations on the portal or on content.
It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.
Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:
def cleanup_wicked_annotation(context=None):
ann = IAnnotations(portal)
if 'plone.app.controlpanel.wicked' in ann:
del ann['plone.app.controlpanel.wicked']
Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:
def cleanup_upload_annotation(context=None):
# remove traces of aborted uploads
ann = IAnnotations(portal)
if ann.get('file_upload_map', None) is not None:
for uuid in ann['file_upload_map']:
del ann['file_upload_map'][uuid]
Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.
The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.
$ ./bin/zodbupdate --convert-py3
You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.
Example:
$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1
Updating magic marker for var/filestorage/Data.fs
Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs
Loaded 2 decode rules from AccessControl:decodes
Loaded 12 decode rules from OFS:decodes
Loaded 2 decode rules from Products.PythonScripts:decodes
Loaded 1 decode rules from Products.ZopeVersionControl:decodes
Committing changes (#1).
After that you should be able to use your ZODB in Python 3.
The process in a nutshell:
bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8bin/instance zodbverify.Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html
See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17
Transforming oids from int to hex and text and vice versa:
>>> from ZODB.utils import p64
>>> oid = 0x2c0ab6
>>> p64(oid)
b'\x00\x00\x00\x00\x00,\n\xb6'
>>> from ZODB.utils import oid_repr
>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'
>>> oid_repr(oid)
'0x2c0ab6'
>>> from ZODB.utils import repr_to_oid
>>> oid = '0x2c0ab6'
>>> repr_to_oid(oid)
b'\x00\x00\x00\x00\x00,\n\xb6'
Get a path for blobs:
from ZODB.blob import BushyLayout
if isinstance(oid, int):
# e.g. oid = 0x2c0ab6
from ZODB.utils import p64
oid = p64(oid)
return BushyLayout.oid_to_path(None, oid)
Load a obj by oid from ZODB in a pdb-prompt:
oid = 0x2c0ab6
from ZODB.utils import p64
app = self.context.__parent__
obj = app._p_jar.get(p64(oid))
Finish, merge and document https://github.com/plone/zodbverify/pull/8
Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:
Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.
This initiative is described in more detail in the Board‘s statement on the Plone governance process. Thanks to the Board for proposing this and we all look forward to a series of fruitful discussions!
Everybody in the Plone community loves sprints. Not only because of the software we enhance or the code we write. The most crucial ingredient of a great sprint is meeting the people you like and that you haven’t seen in a while. This is harder to do in these coronavirus times, but the Beethoven sprint, held remotely from May 27th through the 30th, succeeded.
There were 34 participants, including substantial representation from kitconcept, RedTurtle, and Eau de Web. They used Discord voice channels throughout the day for a real sprint experience. The channels functioned like tables in a sprint room, where sprinters can go from one table to another to hear what people are discussing. The organizers also used Zoom calls for daily standups and Remo.co video calls in the evenings, which are structured like a bar. Sprinters brought their favorite beverage, good food, relaxed on a couch, and had good conversations in the evenings!
In addition to socializing, lots of work got done. The sprint was all about Volto, which is the new React front end for Plone 6. Highlights included:
Thank you to everyone who attended - Plone 6 is getting closer to reality!
PNZ Produkte GmbH is a niche manufactory of oil-based paints and coatings made from renewable resources. Next to its company brand product line, a substantial amount of business is generated from private label (PL) sales. These PL products are offered as a turn-key service: including promotional material, product information sheets and packaging. For some private label customers, PNZ also takes care of fulfillment to end consumers.
A typical finished product is derived from:
Since 2015, Plone has been used as the engine for creating and maintaining product specifications. About 20 employees in multiple departments interact regularly with the software. The project received a “Digitalbonus” innovation award from Germany's Landesamt Oberbayern.
When a new product is formulated, has passed its application and safety testing phase, and is ready to go in production, its specifications will be split in three separate information layers. Custom content types in Plone connect these layers and manage their contents. Plone's ability to quickly prototype content types through-the-web makes it easy to design an initial structure and refine it based on user experiences. Each information layer consists of 30 to 50 individual fields and can be extended as needs change over time. Connecting different types of data from dynamic and external sources is relatively simple as Plone offers a large variety of edit/display widgets out of the box.
At each hierarchical level, new content undergoes a quality and regulations compliance control check before going through the Plone publishing workflow. This makes the document available for use in lower layers. Any base recipe can potentially power several master products, which in turn results in a large number of finished products. The current system powers some 25,000 individual articles or SKU's.
At the lowest level, a "Finished Product" content type includes product-specific overrides and any information not inherited from the recipe and master product layers.
Since products may be offered to multiple geographical markets, all information must be translated. Plone's built-in multilingual capabilities make it possible to have each individual document available in several languages. Fifteen languages are currently in use within the PNZ system. Each field in a content type can be designated as translatable or multilingual, so that updates of multilingual fields can automatically propagate throughout the various translations.
The information contained in its product lifecycle management system allows PNZ to automate the generation of a variety of ready to use content. Plone provides the BrowserView mechanism, used in conjunction with Zope Page Templates (ZPT), to determine what should be included in the content and how it should be rendered. 
PNZ creates colortone samplers and product information sheets, with a customized layout for each brand, using the tools provided by Plone. The documents then undergo a final conversion to PDF using “Produce & Publish” software, created by longtime Zope/Plone community contributor Andreas Jung at Zopyx.
In the packaging process, Plone powers the creative design of all product labels. A BrowserView is used to draw data from multiple content types and the company’s ERP system. This will generate a set of XML files for each product and its SKU variants, plus related dynamic images such as EAN codes, colortones, certifications and GHS warning signs.
The XML files and images are then read into Adobe InDesign templates by the company's creative designers, resulting in a professionally designed wrapper for each SKU that always contains accurate information and complies with industry standards and specifications.
Other types of content include the generation of raw HTML and images for use in a PlentyMarkets shop, and the creation of Product Information Management (PIM) CSV export files for large customers such as DIY store chains. Finally, using plone.restapi, the system powers the product catalog of PNZ's soon to be relaunched public facing website which of course also uses Plone!
June 17-19, Six Feet Up hosted the 2nd Annual Python Web Conference using their newly developed all-in-one virtual event platform, LoudSwarm. Focusing on Python for the Web, the conference featured talks and trainings from several members of the Plone community.
The conference had a total of 266 attendees participating from 37 countries and 14 timezones. A unique aspect of the registration was that for each ticket purchased, a ticket was donated to a developer residing in a developing country. Those that received a free ticket would not have been able to otherwise attend, and they were able to gain valuable knowledge on the state of Plone and Python on the web.
Being a completely virtual event, several accommodations were set up to make the event as interactive as possible. Attendees could join the Gallery after each talk to ask the presenter questions and have discussions about the talk topic. Slack was also heavily used to foster discussions and interaction among the attendees. Evening Socials were created to network, play games, and keep the fun going after the day’s talks were done.
Having a Plone presence was great for the attendees who had never heard of Plone, and provided many with the opportunity to see Plone again in a new way. Many great discussions were had around new Plone features, customization tricks, and fond memories of past Plone Conferences.
Recordings of the talks are currently only available to registered attendees. Post-video access tickets are still available if you weren’t able to attend the live event. All talk and tutorial videos will be made publicly available on YouTube in September.
Plans for Python Web Conf 2021 are already underway. Visit https://2020.pythonwebconf.com to learn more about this year’s event.
Plone was very early to open source governance - the Plone Foundation was one of the first 501(c)(3)s to be established to protect open-source software. In the years that have followed it's become more common for software to be protected under an umbrella organization, but that wasn't possible in the early days.
Zope started off as the property of Digital Creations, which became Zope Corporation. But in the early 2000's there was a desire to move the protection of Zope to a community effort following the same model, so a new non-profit was formed.
This worked well for ten years, but by the mid 2010's there were significantly fewer active developers of Zope and most were not interested in the administrative tasks required to maintain a non-profit corporation. Over the years, the Zope Foundation fell behind in its regulatory compliance until it became inactive. It still owned the copyright to Zope, as well as some funds, but it had very limited rights over them.
The Plone Foundation's mission to protect Plone implies a responsibility to protect Zope, as Plone depends on Zope for much of its core infrastructure. Informal discussions began around this time about Plone taking over the role of the Zope Foundation, but no substantial progress was made.
As the years passed, the need to solve the regulatory problem became more pronounced. It was clear that the Zope Foundation would not restart operations in any meaningful sense, but Zope development was undergoing a resurgence thanks to the need to port to Python 3.
At the 2018 Plone conference in Tokyo, after quite a few whiskies, people resumed talking about this problem, and we decided to try to solve it again. Because one of the biggest blockers was a lack of enthusiasm in the Zope community with regards to dealing with the bureaucracy, some people who were members of both Foundations reached out to the IRS, the Delaware Department of State and some experts in US Corporation law to plan a way forward.
A few months later in early 2019, the Zope Foundation held a special general meeting to pass a motion on the future of the Foundation. The membership mandated the board of directors to transfer all assets to the Plone Foundation as a donation, and to formally close the Zope Foundation. Later that same day the board of directors voted to donate all property of the Zope Foundation, including copyright in the Zope software, to Plone.
Since then, Plone community members have been working on the bureaucratic aspects of this move. We've harmonized contributor agreements to improve international enforceability, and work is underway to ensure that Zope contributors are properly represented in Plone's democratic structure before the next internal election in November. Zope contributors can already apply to be members of the Plone Foundation, and many are, but we're looking to streamline the process for former Zope Foundation members.
The Zope and Plone Security Teams have already been integrated, and now we need Zope representatives on the Membership Team. Any Zope people who are interested in helping out should please contact Érico Andrei.
July 19, 2019, marked the day Plone officially entered the world of Python 3 when version 5.2 was released. Support for Python 3.6, 3.7, and 3.8 is now included with Plone 5.2, opening up the future for Plone’s continuing 19-year reputation for unrivaled security and stability.
Plone is the venerable open-source Python web content management system that was created in 2001. It has continued to be maintained and enhanced by hundreds of contributors around the world and is protected by the nonprofit Plone Foundation.
Plone 5.2 is the culmination of over 3.5 years of development that was driven by the looming deadline of January 1, 2020: the impending end of support for Python 2. Work on Plone 5.2 ran in parallel with continuing feature-centric work on versions 5.0 and 5.1. The move to Python 3 was to be a major undertaking, given its huge codebase, its large ecosystem of add-ons, and its dependency on the Zope web application framework (https://www.zope.org), which itself would have to be upgraded to Python 3. Many sprints were held over these past few years in which Plone and Zope developers collaborated in this joint effort. One of the earliest fruits of this collaboration was to see all 9000 tests passing as a result of work done at the Alpine City Sprint in February 2017, but more work was still needed. At the core of the security model for which Zope and Plone are renowned is a feature known as Restricted Python. Although core developers had feared that Restricted Python could not be ported to Python 3, this was accomplished in Innsbruck, Austria: Restricted Python now runs in Python 3.6 and up.
ZODB, the ACID compliant and extremely robust database, was upgraded to the latest version for Plone 5.2, along with a Zope instance running via a WSGI server instead of ZServer.
Zope developers came together and released the Python 3 compatible Zope 4 in May 2019. Plone developers were able to release its Python 3 compatible version 5.2 soon after, complete with its traditional in-place upgrade, with an additional database upgrade step required to convert to Python 3 strings. Plone 5.2 is the first release to fully support Python 3 while maintaining complete compatibility with Python 2.7, which is expected to be dropped with the upcoming release of Plone 6.
Once available as a separate add-on, a REST API is now also included in the core of Plone 5.2, making it even easier to use Plone with modern JavaScript front ends such as Angular and React and to integrate Plone with other applications.
This release provides the Plone community the certainty of having a stable backend without an expiration date and the excitement of developing a new frontend. Plone is more than 19 years old, and Zope is more than 22 years old. Yet both projects have reinvented themselves several times and are maintained by a worldwide, dynamic community. These two platforms provide a very secure, flexible, and mature solution in the often uncertain, unstable landscape of web applications.
Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.
Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.
Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.
Now they had the specification, but how did relying on Plone help with the integration?
To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.
In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.
The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.
Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!
So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.
Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:
For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.
RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…
For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.
The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.
In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.
Welcome our full subtitle automation story:
Profit.
An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!
In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.
At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.
As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.
But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.
After all, a real CMS was required.
Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.
And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.
The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:
,
All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:
What you see, edit and connect in Volto…
…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.
Something old, something new and something blue.
The perfect match.
Last week we held the first Plone remote sprint in the time of the coronavirus pandemic. It was a positive experience and a lot got done! Sprint plans and results were captured on the plone-6-sprint Etherpad and some chatting occurred on the plone/sprint Gitter channel. But the main method of communication was by video. Participants stayed connected by video all day, either on Jitsi or Discord. Each team had their own "room", and it was nice to be able to see everyone while working, even when microphones were muted. This mimicked the sort of frictionless in-person conversations that are the hallmark of sprints.
The sprinters tackled six different topics, here are the final reports on each. For more detailed information, see the plone-6-sprint Etherpad.
Peter Holzer and Stefan Antonelli led the charge with help from Robert Kuzma, plus Peter Mathis taking the lead on z3c.form. The team made a long list of contributions toward the following PLIPs:
Their work changes the default theme plonetheme.barceloneta to be based on Bootstrap 4 and SCSS. Plone specific styles have only been added where Bootstrap is lacking definitions for Plone elements. There is an overrides directory where all template overrides will be collected until they are merged back into the original packages.
Markup in general was improved. main_template markup was updated and the structure cleaned up, empty and unneeded elements have been removed. Header, navigation, portlets, status messages, batching - all markup and nav styles were updated. The generic view for content types was updated (this includes title, description, byline, related items and keywords). control_panel_overview, control_panel default, and some control panels with custom views were all updated.
For forms, Bootstrap markup was added to plone.app.z3cform to give widgets and fields (buttons, input fields, etc.) a basic styling. Plone's classic backend forms are also fully functional as long as Bootstrap's default CSS is independently available. In fact a theme (e.g. Barceloneta) is not necessary as long as Bootstrap CSS is available.
Icons also got some love. icon_expr for actions, content types, etc. was reused to define SVG icons. bootstrap-icons were added as an available icon resource to plone.staticresources. The team selected and added icons to the control panel settings in the controlpanel.xml in plonetheme.barceloneta.
Mosaic was changed to use Flex for columns based on the updated Barceloneta theme. All LESS files were replaced with SCSS and 'npm run watch' is now used to compile the SCSS files. Redundant CSS classes were cleaned up and the default column limit was set to 6 columns. A reset button was added to Mosaic cells to reset to the automatic column width, and an automatic reset was added for column widths if a new column gets inserted. A label was added to indicate the current column width, with "0" for auto width.
A demo of classic Plone 5.2 with the latest Barceloneta default theme is at demo.plone.org. The Coredev buildout including the current state of work on the PLIP for Bootstrap 4 markup is at classic.plone.de.
The team plans to meet regularly to continue working every Wednesday at 5 PM CEST starting on May 13. They will meet in the plone-6-sprint-barceloneta Jitsi room. Everyone is welcome to join!
Maik Derstappen led this project, in coordination with the Barceloneta work. They added a custom.css view, which renders the custom CSS added in the Theming Control Panel, after all other CSS resources are loaded. This allows you to override CSS coming from a Diazo theme.
The team also discussed what really needs to change in the Resource Registry. They decided to remove the build-button from the theming editor because it will not work with SCSS (which is now used by Barceloneta).
Developers should register resources already compiled as Resource Bundles with TTW compiling disabled. They can use the merge flag to merge it with default or logged-in bundles, but this only makes sense if they are always needed. Otherwise the resource can be disabled and loaded on demand via Python in the view/viewlet or by adding the header lines directly into the templates.
Variables can be defined as CSS variables. Variables from the registry are no longer used in SCSS/CSS in plonetheme.barceloneta.
Jens Klein reports that lots of "boring but important work" was done by Maurits van Rees including cleanup of package metadata and releases of packages. Jens Vagelpohl released Zope 5.0a2 and Jens Klein created a zope-5 branch for Plone 6.0 on buildout.coredev. Version dependencies were updated on this branch, and version pins that are orphaned or handled upstream by Zope were removed. Version dependencies still outstanding:
Many failing tests were fixed. Four tests (partly related to zope.interface 5.1) are still open and in-progress. Jens would appreciate help to get them fixed and merged.
Timo Stollenwerk led the Volto efforts with Victor Fernandez de Alba, Rob Gietema, Steffen Ring, Katja Süss, Alok Kumar and Rodrigo Ferreira de Souza. They did a complete overhaul of the folder contents view according to Albert Casado's Pastanaga UI design, and made numerous other UI improvements based on a full UX review that Albert did. Alok (frontend) and Rodrigo (backend) implemented the Add-ons Control Panel and Alok implemented the Users and Groups Control Panel. Alok also implemented print CSS for the Volto front end. Victor fixed the listing block to remove the current item from query results for ZCatalog 5.1. Finally, Steffan and Katja worked on content for the upcoming VoltoCMS.com website.
Asko Soukka worked on image scaling in coordination with the Volto team, with additional help from Jens Klein and Alexander Loechel. They fixed several bugs:
They implemented an option (disabled by default) to generate all configured scales immediately once an image has been updated. Enabling this would slow down the save of an image field, but would prevent most writes on read for image scaling. The code is in plone.formwidget.namedfile.
They implemented a proof of concept for PLIP #3090 for asynchronous multiprocessing of image scaling, which compliments the above option to generate all configured scales on save. It could provide image scaling with good performance for Plone without any additional services.
They implemented a proof of concept for the use of picture & srcset in Volto for perfectly responsive images. The code is in Asko's Github account.
Philip Bauer led a team consisting of Steve Piercy, Katja Süss and Janina Hard to create and update the developer training class for Plone 6. Much was accomplished. The first alpha version of chapters 1 through 36 are now usable and will be deployed soon for testing.
Many of the chapters now exist in two flavors: one for Volto and one for Classic, with links between them. Logos at the top clearly identify which flavor you are reading. The default navigation follows the Volto story.
The team plans to do bi-weekly half day sprints and to finish the training at the Beethoven Sprint which is scheduled for May 26th. The outstanding chapters are:
Sally Kleinfeldt and Érico Andrei investigated tools and planned a series of "how to" screencasts for the Plone Youtube channel about managing content in Plone. Érico also worked on adding the videos from the 2003 to 2009 Plone conferences to the channel.
Sally discussed how to position headless Plone and Volto with Timo Stollenwerk, Victor Fernandez de Alba and Katja Süss. As noted above, the Volto team is working on a new voltocms.com website which will primarily be aimed at React developer on-boarding. We discussed how plone.org/plone.com can coordinate with and compliment their efforts.
After discussion with additional interested parties (Rikupekka Oksanen, Stefania Trabucchi, Fulvio Casali and William Fennie), we decided to create a "What is Plone?" section on plone.org which is aimed at decision makers. It will give a bit of Plone history and describe the current variety of frontend and backend options and when to use them. We began brainstorming ideas and will meet again in 3 weeks. Input from other community members would be welcome.
“How hard can it be? It is just a theme…”
Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.
Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.
Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.
Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.
We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.
Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…
The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.
Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.
With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.
All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.
When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.
Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).
In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.
At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.
Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:
Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.
All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.
The COVID-19 pandemic has upended life across the planet. Scientists, doctors and other experts are engaged in a giant global effort to combat the disease. Open source software is integral to this effort because many tools and libraries used by these experts are open source. That includes Plone, a great platform for building secure, scalable, highly customizable, content-rich websites for scientists and researchers.
One such Plone site is already being used in the COVID-19 fight. The Onkopedia portal publishes medical guidelines for oncology and hematology for Germany, Austria and Switzerland - treatment guidelines, medical studies, protocols, certifications, drug information, etc. for over 60 diseases. Patients with these diseases are at high risk for the new coronavirus, which created an urgent need for providing COVID-19 specific information. In just 3 days the Onkopedia development team was able to extend the site's data model and roll out necessary UI updates to create a new Onkopedia section with COVID-19 content for over 40 different diseases. Congratulations to the team for a great example of agile development, and to Plone for providing a flexible and robust platform.
Do you have a success story about Plone helping to fight the pandemic? Let us know at [email protected].
After months of planning, in early March the PLOG organizers had to face the reality that traveling to Italy in April was not going to be feasible. We tried to reschedule the event in September, but the hotel was already booked up. So we've had to make the sad decision to cancel PLOG for 2020.
PLOG will return in 2021!!!
For the foreseeable future, Plone community sprints will need to happen remotely - individual sprinters in their own spaces, communicating via audio, video, and text. Happily, the big April hole that PLOG's cancellation left in the Plone calendar has now been filled by a remote sprint on Plone 6. Kudos to Peter Holzer, Maik Derstappen and Jens Klein for organizing it! The sprint will tackle a number of topics:
Every experience level is welcome, and additional sprint topics are welcome too. Sign up and add your ideas!
These days are tough and challenging for all of us. COVID-19 affects everyone across the globe. The Coronavirus hits every country differently. Fortunately, Germany seems to be better prepared than other countries. However, there will be a huge fallout from this crisis and many things will not be the same as they were for a while. So, please #stayathome and stay safe.
But let me tell you a small success story around Plone and COVID-19.
We are running the portal Onkopedia (www.onkopedia.com) for over a decade where the medical guidelines for oncology and hematology for Germany, Austria and Switzerland are published. The portal contains treatment guidelines for over 60 diseases which also includes medical studies, protocols, certifications and information about individuals drugs and active components. With the growing Corona crisis in Europe, high-risk patients with pre-existing conditions like cancer came into the focus. There was an urgent need for providing COVID-19 specific information for most diseases to the public.
The Onkopedia portal was always running and powered by Plone (nowadays running on Plone 5.2, Python 3). Within two or three days we had to extend our data model for COVID-19 in order to provide the functionality required to publish the COVID-19 information specific for each disease. With the help of the whole Onkopedia development team, we were able to to roll out the necessary Onkopedia update within a very short time from of three days. The required changes affected the Plone data model for the guidelines and a bunch of UI aspects like a new COVID-19 section and individual markers in guideline documents for referencing the related COVID-19 information in the COVID-19 master guideline.
As a result, we managed to roll out the new COVID-19 section on onkopedia.com (German only) yesterday with individual COVID-19 content for over 40 different diseases.
Key success factors: Plone and a great team (Abstract Technology, Practice Innovation, AppWeeve)
Thank you all for making this happen and for contributing in the fight with COVID-19.
Each year, the Plone German-speaking Community organizes a 3 day Symposium (Plone-Tagung 2020) and a two day sprint for German-speaking universities, NGOs and companies as well as for users, service providers, and developers.
The symposium this year took place at the Technical University of Dresden (TUD) from the 9th to the 13th of March, 2020. The Symposium was supported by the Plone Foundation and sponsored by Werkbank (Gold Sponsor), Zopyx and FlyingCircus (Silver Sponsor) and Abstract-Technology (Bronze Sponsor).
The Plone Symposium in Dresden focused on the opening keynote from Anne-Marie Nebe on accessibility and why web accessibility is so important for user-centered software development. The Humanities and Social Sciences Department of the TU Dresden contributed with their expertise on the linguistic and visual implementation of a barrier-free web presence. The second-day keynote "Chaos und Diversity“ from Christian Theune illustrated the characteristics and dynamics of complex systems and talked about new approaches for a conscious use of diversity.
A complete overview of all topics of the Symposium is available, in German, at the Plone Tagung website.
Building an open-source community isn’t easy but now you can print your own Plone cookie cutter and make these goodies on your own! https://www.thingiverse.com/thing:4226416
Thanks to @quirk_dispenser.
A big thank you goes to the Technical University of Dresden (TUD) for hosting, the Plone-Team TUD (@plone_tudresden) for organizing the event, and to the speakers, sponsors and all participants, who made the event special.
If you weren’t able to make it this year, we hope you’ll join the Plone Tagung next year! See you then.
Volto 4.0 is now ready, following a 9 month alpha period that included many sprints.
A lot of effort has been put into keeping Volto's learning curve low and ensuring that developing with it is a delightful experience.
Here are some of its new features:
* Improved Pastanaga editor
* New Pastanaga editor sidebar
* New mobile first toolbar
* Developing blocks is now easier than ever
* New object browser
* Listing, table of contents, and lead image blocks
* New blocks chooser and future-proof blocks definitions
* Body classes like Plone's hinting at content types, section and current view
* New message system
* React hooks support
* Several internal libraries updated, including Redux, Router ones that support hooks
* Many bug fixes
For further details, please see the Volto 4.0 announcement forum post
TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.
Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.
The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.
So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:
At first ./buildout.cfg for the Plone instance:
[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions
extensions = mr.developer
sources = sources
auto-checkout = *
[sources]
transmogrifier = git https://github.com/collective/transmogrifier
[instance]
recipe = plone.recipe.zope2instance
eggs =
Plone
transmogrifier
user = admin:admin
[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instanceThen buildout must be run to create the instance with a Plone site:
$ buildoutNext the transmogrifier ./pipeline.cfg must be created to define the pipeline:
[transmogrifier]
pipeline =
from_rss
prepare
create
patch
commit
[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot
[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')
[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)
[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))
[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):
$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSiteThis example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.
There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.
In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)
At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:
[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions
extensions = mr.developer
sources = sources
auto-checkout = *
[sources]
transmogrifier = git https://github.com/collective/transmogrifier
[instance]
recipe = plone.recipe.zope2instance
eggs =
Plone
transmogrifier
plone.restapi
user = admin:admin
extra-paths = local
zcml = custom
[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instanceBefore running buildout we ensure a proper local Python module structure with:
$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.pyOnly then we run buildout as usually:
$ buildoutNow, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:
# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint
import json
import pathlib
class Glob(Blueprint):
"""Produce JSON items from files matching globbing from option `glob`."""
def __iter__(self):
for item in self.previous:
yield item
for p in pathlib.Path(".").glob(self.options["glob"]):
with open(p, encoding="utf-8") as fp:
yield json.load(fp)
class Folders(Blueprint):
"""Minimal Folder item producer to ensure that items have containers."""
def __iter__(self):
context = self.transmogrifier.context
for item in self.previous:
parts = (item.get('_path') or '').strip('/').split('/')[:-1]
path = ''
for part in parts:
path += '/' + part
try:
context.restrictedTraverse(path)
except KeyError:
yield {
"_path": path,
"_type": "Folder",
"id": part
}
yield itemAnd complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:
<configure
xmlns="http://namespaces.zope.org/zope"
xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">
<include package="transmogrifier" file="meta.zcml" />
<transmogrifier:blueprint
component=".blueprints.Glob"
name="custom.glob"
/>
<transmogrifier:blueprint
component=".blueprints.Folders"
name="custom.folders"
/>
</configure>Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:
[transmogrifier]
pipeline =
generate_from_json
generate_containers
set_container
create_folder
create_document
commit
[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json
[generate_containers]
blueprint = custom.folders
[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])
[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])
[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
plone.api
plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))
[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()Finally, the pipeline can be run and content imported with:
$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSiteObviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.
If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.
Technology never stands still.
It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.
A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:
This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.
Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.
Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.
The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.
Users will see:
Editors and admins will see a number of new features under the hood:
Website developers will be able to use:
Visit Plone.com to read more about the advantages of Plone 5.
The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.
Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.
The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:
These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.
Now that’s worth upgrading for.
The Plone Foundation Board has appointed these outstanding individuals from around the world to serve as ambassadors for Plone.
They are known for their community involvement in their regions and for their expertise in a particular market of Plone.
In their capacity as Plone ambassadors, they will continue to promote Plone regionally and in their specific markets for Plone.
Plone's ambassadors for 2020 are:
See https://plone.org/foundation/board/ambassadors for more information
"Flags of member nations flying at United Nations Headquarters". 30/Dec/2005. UN Photo/Joao Araujo Pinto.
If you use Waitress, please upgrade from 1.4.2 to 1.4.3.
The Pylons Project released a new version of Waitress to fix a bug in the regular expression that was used to parse HTTP headers. The bug could cause the waitress process to use excessive CPU.
As Plone 5.2.1 uses Waitress 1.4.2, we recommend changing the version pin to 1.4.3 in your buildout.
[versions] waitress = 1.4.3
As announced previously, the 20200121 hotfix includes several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting.
Version 1.1, released on February 11, 2020, includes an update for the SQL Injection fix, which will not be needed for all installations.
If you are not using SQL in your website you do NOT need to upgrade, though you can if you want to. Default Plone does not need it. Upgrading to this version is especially recommended when you use PostgreSQL. Note that RelStorage is not affected. For details and discussion, see DocumentTemplate issue #48.
Full installation instructions are available on the HotFix release page.
These are standard precautions that should be employed on any production system, and are not tied to this fix.
If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers
There is also free support available online via the Plone forum and the Plone chat channels.
Q: When will the patch be made available?
A: The Plone Security Team released the update patch on 2020-02-11T22:47:25+0000.
Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.
Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.
Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.
Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.
Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.
Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?
A: Details about the vulnerability will be revealed at the same time as the patch.
Q: How can I report other potential security vulnerabilities?
A: Please email the Plone Security Team at [email protected] rather than publicly discussing potential security issues.
Q: How can I apply the patch without affecting my users?
A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html
Q: How do I get help patching my site?
A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels
Q: Who is on the Plone Security Team and how is it funded?
A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.
Q: How can I help the Plone Security Team?
A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors
General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at [email protected]
To report potentially security-related issues, email the Plone Security Team at [email protected] We are always happy to credit individuals and companies who make responsible disclosures.
We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.
PLOG is a unique sprint where you can spend the mornings training, the afternoons sprinting, and all day enjoying the southern Italian weather and food. Thank you to the Plone Foundation, which has provided funding so that we can offer the following FREE training classes. It's a great way to brush up some old skills and pick up new ones at the halfway point between Plone conferences.
Thank you to the Plone Foundation, which is providing funding to support the training classes.
This class will be based on the Volto training given at the last Plone Conference. It assumes some knowledge of React (but you don't have to be an expert.)
This class will teach you how to deploy a static website from Plone content, and how to integrate it with the GatsbyJS ecosystem (plugins, themes, cloud services, etc.)
This class will cover a number of advanced Plone developer topics, including:
Because of its importance to the community, PLOG has been designated a strategic sprint by the Foundation Board and the Marketing Team. A wonderful cross section of people will attend, so there will be sprinting activities for all interests. Some will be working on Volto and RestAPI improvements, others will be creating marketing collateral in the form of "how to" screencasts of Plone features. Bring your own topic - the final plan will be made at the sprint.
The registration deadline is February 28th, after that we cannot guarantee the discounted rates or room availability.
Read all about the Plone Open Garden, a tradition in our community since 2007. There's even a PLOG video. Dates, prices, and all of the details are given on the PLOG 2020 event page.
CVE numbers: CVE-2020-7936, CVE-2020-7937, CVE-2020-7938, CVE-2020-7939, CVE-2020-7940, CVE-2020-7941.
Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.
Versions Not Affected: None.
Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.
The patch was released at 2020-01-21 15:00 UTC.
Full installation instructions are available on the HotFix release page.
These are standard precautions that should be employed on any production system, and are not tied to this fix.
If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers
There is also free support available online via the Plone forum and the Plone chat channels.
Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2020-01-21 15:00 UTC.
Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.
Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.
Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.
Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.
Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.
Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?
A: Details about the vulnerability will be revealed at the same time as the patch.
Q: How can I report other potential security vulnerabilities?
A: Please email the Plone Security Team at [email protected] rather than publicly discussing potential security issues.
Q: How can I apply the patch without affecting my users?
A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html
Q: How do I get help patching my site?
A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels
Q: Who is on the Plone Security Team and how is it funded?
A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.
Q: How can I help the Plone Security Team?
A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors
General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at [email protected]
To report potentially security-related issues, email the Plone Security Team at [email protected] We are always happy to credit individuals and companies who make responsible disclosures.
We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.
CVE numbers not yet issued.
Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.
Versions Not Affected: None.
Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.
The patch will be released at 2020-01-21 15:00 UTC.
This is a pre-announcement of availability of this security fix.
The security fix egg will be named Products.PloneHotfix20200121 and its version will be 1.0. Further installation instructions will be made available when the fix is released.
These are standard precautions that should be employed on any production system, and are not tied to this fix.
Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers
There is also free support available online via the Plone forum and the Plone chat channels.
Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2020-01-21 15:00 UTC.
Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.
Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.
Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.
Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.
Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.
Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?
A: Details about the vulnerability will be revealed at the same time as the patch.
Q: How can I report other potential security vulnerabilities?
A: Please email the Plone Security Team at [email protected] rather than publicly discussing potential security issues.
Q: How can I apply the patch without affecting my users?
A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html
Q: How do I get help patching my site?
A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels
Q: Who is on the Plone Security Team and how is it funded?
A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.
Q: How can I help the Plone Security Team?
A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors
General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at [email protected]
To report potentially security-related issues, email the Plone Security Team at [email protected] We are always happy to credit individuals and companies who make responsible disclosures.
We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.
Training classes on several topics will be held in the mornings, sprinting will happen in the afternoons and discussions will go on all day. The Hotel Mediterraneo is in a beautiful location overlooking the Bay of Naples, with wonderful food and a lovely garden where the sprinting and training will be held.
We have reserved a limited number of rooms and they are available on a first come first served basis. Breakfast and dinner are included. The rooms for three and four persons are the most economical and are expected to sell out early.
The Plone Foundation welcomes four new members, after unanimous confirmation by the Foundation's Board of Directors on October 10, 2019.
Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.
Fulvio started working with Plone in 2008 and since 2012 he has been operating a consultancy business, Soliton Consulting, specializing almost exclusively on Plone. Fulvio organized Emerald Sprints (2013, 2014), Plone Open Garden (2016, 2017, 2019, 2020), participated in every Plone Conference since 2010 (except one!), has given two conference talks, and attended the Plone Symposium East 2010 and the Plone Konferenz in 2012. Fulvio has provided Plone training for Mastering Plone, Diazo theming, Rapido, Angular, and TTW Dexterity.
Stefania is the co-founder of Abstract Technology in Germany. She was the organizer of Plone Meet Up Berlin 2004-2014, of World Plone Day Berlin 2009-2015, of Plone Social Sprint Berlin 2014, and of Plone's Europython 2014 presence and associated marketing. She has participated in the Plone Konferenz 2012 as a speaker, the Plone Beethoven Sprint 2019, the Plone Conference 2014 sprint, and the Plone Open Garden sprints in 2014 and 2015. Stefania represented Plone as a member of the CMS Garden 2014-2017 and is active in the Plone Intranet Consortium with Quaive.
Thomas is the CTO of 4teamwork AG, has been working with Plone since 2006 and became a code contributor in 2011. He has attended many Plone conferences, beginning with Naples in 2007. He attended many sprints, including the Barcelona Strategic Sprint (2016), the Bonn Beethoven Sprint (2017 and 2018), the Sorrento Sprint on Frontend Modernization and Python 3 Porting (2019), and the Beethoven Sprint (2019). Thomas' main code contributions have been in the plone.rest and plone.restapi modules.
Andrea was the lead organizer of the Plone Conference 2019 in Ferrara. He participated at earlier Plone conferences and sprints (Sorrento 2019, Tokyo 2018, Barcelona 2017, Bristol 2014). As part of the RedTurtle team, he has organized and participated in World Plone Day events since its inception. Andrea is the maintainer of over 100 packages on PyPI.org.
The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.
The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.
If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.
Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.
Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.
Following the election held in October 2019, the 2019-2020 Plone Foundation Board held its first meeting on November 7, 2019, and its first order of business was to select its officers:
The Board also selected the following committee chairs:
The following (unofficial) team liaisons were selected:
The new Board is determined to reflect the changing landscape of the Plone Foundation, where it is not just about the Plone CMS but a host of technologies and communities that reflect the same spirit, including Volto, Guillotina, Zope and others.
The Plone Foundation Board is now in its 2019-2020 term.
The results of the Foundation Membership vote were announced at the Annual General Meeting held in Ferrara on October 25, 2019.
The 2019-2020 Board members are:
William Fennie was not elected although he received over 30% of votes.
Our outgoing Board members chose not to run this year. We thank them for their many years of service!
Demo of tool for online risk assessment. Export to Word (with python-docx). Training module. Export to Powerpoint presentation with python-pptx.
State of Plone Intranet in 2019. Dozens of packages. Social stream, search, auditing, real-time document collaboration. Classifieds app makes it nice to share info, legal app showing contracts. PAS graph optimizations, see talk by Alessandro. SVG rendering. Progressive web app, we did not want to create a special app, but it does really give a mobile experience.
Busy supporting Plone 5.2 and Python 3, we will manage that. We will use crossbar.io for push notifications. Encryption via CWS. Johannes is joining Syslab to work on Quaive fulltime. iMio is helping too.
The last of my three easy pieces on Ferrara.
vini delle sabie is a wine. Sand is influencing the grapes. The grapes are actually immune to some diseases. Most typical is de fortana grape. Give it a try. Perfect companion to the dishes I talked about.
Example package with LDAP setup. Uses pas.plugins.ldap, which is a replacement for plone.app.ldap. The core package now supports wildcard search, with help from Asko Soukka. Robert Niederreiter did a lot of work to get it working on Python 3. LDAPS support.
Desktop document generation (.odt, .pdf, .doc, ...) based on appy framework and OpenOffice/LibreOffice. Templating in LibreOffice.
Code: https://github.com/collective/collective.documentgenerator
Create news letters in Plone. Send mails to subscribers. Let email server do this, or use an external delivery service. On a collection you can define how items are rendered in the email. You can customize the output and aggregation template. Or fully write your own. Works now on Plone 5.1/5.2 Python 2.7/3.7.
Planned: integrate Mosaico editor.
9 to 11 March 2020 in Dresden. Talks and sprints. Come!
We plan to create quickstart Plone trainings for Python devs, one for frontend devs, one for users. Mastering Plone 5.2 talks about a lot, you should follow it.
But what about mastering Plone 6. It would need to be about backend and frontend. I would love feedback about what would be required. We do not want to overwhelm people.
I present three PAS plugins:
See the full presentation.
There was a treasure hunt this week, where every day you had to find three items in the city and make pictures. We show the solutions.
We use this Acceptance testing framework to test Volto. It has gained traction in the JavaScript world.
I use robotframework and love it. But Cypress is written in JavaScript, so an intern could write extra plugin library for it.
The next two days we will have Plone sprints. Everyone is welcome. It is a great chance to share code, get to know the Plone code, ask questions to coders, and generally work together. We will find someone to pair up with you.
See the list of sprint topics. You can work on other stuff, you can add ideas, also when you will not work on it. You can add your name to topics when you are interested, also multiple topics.
People could suggest add-ons, and then people voted for them. Clear winners:
Others:
The Plone conference next year will be in... Namur, Belgium. November 16-22.
Thank you Red Turtle for organising such a lovely conference!
Plan:
For each system:
Discussion:
Use cases:
Pyramid:
Django:
Guillotina:
Plone:
Three PAS plugins:
Deny access to roles like Manager and Editor
PAS plugin for authentication based on request headers.
Use case: Apache/nginx adds SAML headers to requests.
Configuration in ZMI or profiles/default/pas.plugins.headers.json:
{
"userid_header": "uid",
"required_headers": ["uid"],
"roles_header": "roles",
"allowed_roles": ["student", "teacher"],
"deny_unauthorized": true,
"redirect_url": "https://maurits.vanrees.org",
"memberdata_to_header": [
"fullname|HEADER_firstname HEADER_lastname"
]
}
Plone PAS plugin for content as groups.
Code: https://github.com/collective/collective.contentgroups
What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.
It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.
There are about twenty different plugin types, and they interact with each other.
In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active
So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.
Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)
We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.
We wanted collective.workspace to manage roles through groups instead of sharing.
And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.
It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.
So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.
So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.
And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.
There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.
Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.
Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.
To keep the data structure up to date, we use events.
To recap:
Further ideas:
Takeaways:
At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.
Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.
Clients:
Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.
We use Jenkins to test the migration.
General backend things:
What we polish to enter in Volto land:
Question: can you open source the module that handles the RichText to Volto blocks migration?
Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.
See the annual report.
Jen Myers has graciously offered to stay on next year as treasurer.
The Zope Foundation has been added to the Plone Foundation. Still ongoing, complex process.
With the Pylons community, those talks have stalled at the moment. Might happen in the future.
Financially we lost more money than we usually do. There was a trademark conflict that cost a lot more than we thought.
Some money should come from the Zope Foundation by the way.
Results of the vote for the board.
48 valid votes, one invalid. 2 went to the spam folder, but we found them. 30 percent of votes were in paper.
We had a vote by the foundation membership. Voted into the Plone board have been: Victor, Erico, Chrissy, Andy, Paul, Jens, Fulvio.
Meeting closed by Paul.
Thanks to Alexander, Kim and Carol who are leaving the board.
We wanted to find an easy way to reproduce the installation process. We use:
Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.
Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.
More libraries with Fabric:
Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.
I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.
I have some upgrade code that you can use, or for some parts copy and adapt for your use case.
Code: https://github.com/collective/collective.migrationhelpers
Upgrade steps:
You may want to disable LDAP temporarily in an upgrade step.
Get the birds eye view, with some code that reports:
Divide and conquer:
You can register upgrade steps conditionally in zcml if needed, for example with zcml:condition="installed plone-52".
Make big problems small:
Forget the past:
Migrating LinguaPlone:
Update to Plone 5.2:
Archetypes to dexterity:
Alternative: inplace migrator from ftw.upgrade. Interesting for large folders. Might be a nice PR to get this part into core.
To Python 3:
Why do it this way? I don't want every Plone company to have their own migration code stashed away somewhere. Please contribute to the core.
Upgrade to Plone 6? Install plone.restapi, use Volto, done.
A year ago, ZServer was one of the last Zope parts that did not run on Python 3. The idea was to use WSGI instead. I wondered why Python 3 would not be possible. So I tried porting it. It worked. And I added websocket support. You can use this to automatically push pages from Plone to Gatsby. Or show a notification when a page needs review, via content rules.
I replaced the old Medusa server with Twisted. ZServer was optionally using Twisted years ago, but this was ripped out, and I don't know why. But I got it working again. So for HTTP and webdav we have a Twisted http server and thread pool with an async FileSender, an AsyncIO/uvloop when avilable, standard ZConfig configuration and logging, And HTTP/2 works as well.
Bonus: websockets with PubSub messaging. It uses the Autobahn Twisted WebSocket protocol. ZeroMQ PubSub cockets using IPC-socket. The connection autosubscribes to local events. The server publishes a guarded event, for example indicating that an event is restricted to Editors.
One problem at this moment with the wsgi setup using waitress, is that the requests are limited to the number of wsgi threads, so server large files can become problematic. There are probably ways around this, but ZServer does not have this problem.
Status:
Upstream:
Remaining steps:
I would be interested to hear when others are interested in using this. It is better if more people use this.
I am an introvert from Finland, so I did not ask the Zope people, or visited the Zope sprints.
Jens: There was no real reason to drop ZServer, except that no one has tried it because the code was so very old. Now you got it working, let's get this into Zope.
See the slides.