Plone Logo

Planet Plone - Where Developers And Integrators Write

Open Source Week 2020 Recap

Posted by PLONE.ORG on January 05, 2021 08:50 PM

Open Source Week 2020 was held from December 1 to December 4. During the event, several topics about the Open Source world were discussed. Four storytelling days about Open Source through the most reliable voices in this world, the customers' feedback, technical workshops and focus on products and solutions.

https://opensourceweek.it.

There was talk about enterprise business, the complexity of development models and business, the ethical value, and the legal and security challenges from the perspective of the top insiders. The first entirely online RIOS conference dedicated to the Open Source world dealt with Cybersecurity, Collaboration, Machine Learning, and Artificial Intelligence, Cloud Native, DevOps+Sec, Development, IoT, Augmented Reality, Industry 4.0, BigData, Analytics, all in an Open Source key.

As part of World Plone Day on December 3rd, we talked about Volto and the new developments of Plone.

Plone 6 example

The event in its 4 days has brought the participation of about 500 people and the sale of more than a thousand tickets. Through sponsored posts on social media, the participation of the event was maximized, in addition to targeted posts during the conference on the main social channels. Through the social campaigns, Plone's posts were seen by more than 18 thousand people.

The Plone logo has been mentioned as an Official Sponsor on the event website https://opensourceweek.it.

Going Wordpress

Posted by Andreas Jung/ZOPYX on January 05, 2021 11:26 AM
Going Wordpress

I am moving corporate sites to Wordpress  🤡 -t his is not an April 1st joke!!!

I am currently in progress moving some of my corporate sites and some project sites from Plone to Wordpress. I struggled a long with the decision what to do with my almost ten year old corporate sites. I spend almost two year into static site generators, custom HTML page builders, frameworks like Tailwind UI kit and many more.

I always had an eye on Wordpress of the last years but stability never convinced me. In fact, I could break a fresh Wordpress within minutes...but this has not happened yet with Wordpress 5.6.

As a proof of concept and for getting deeper into Wordpress, I created a new project site (not online so far) for checking various site builder options, extensions and add-ons and deployment options.

Here is my current stack:

  • Wordpress 5.6 (installed and maintained through the wp commandline utility)
  • Porto theme
  • Elementor Pro as site builder (feels super natural compared to WPbakery)
  • Smart Slider 3
  • UpdraftPlus for backup and restore (stores all data on Dropbox)
  • WP Mail SMTP for mail
  • Caldera Forms - an super intuitive form builder
  • miniOrange 2 for 2FA
  • FileBird Lite - for folder support under "media"
  • DSGVO AIO for imprint/privacy and cookie alerts

My current Wordpress 5.6 setup appear rock solid and it just doing the thing. "The thing" is creating (more or less) shiny websites and focus on content. The available theme for Wordpress leave a lot of room for your own corporate identity. I do not care so much about tiny design details and I don't want to invest a lot of time and money into my "own" theme.

In the end, the decision was based on "getting things done" in a reasonable timeframe and for a reasonable investment.  Of course, Wordpress leaves too many styling options for content creators but that's exactly what Wordpress is for in the first place: building shiny corporate sites, landing pages etc. This is not primary scope of Plone and the primary scope of Plone can usually not be covered by Plone. So there is no contradiction here using Wordpress as a Plone solution provider.

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

Posted by Maurits van Rees on January 05, 2021 10:33 AM

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o <oid> -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o <oid> -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.

Plone Conference 2020 Recap

Posted by PLONE.ORG on December 20, 2020 04:54 PM

Plone Conference 2020 Online

Plone Conference 2020 was a 9-day event for the Plone, Zope, Volto, Guillotina & Pyramid communities and consisted of training, talks, and sprints. The conference provided insight into the long history of Plone CMS as well as the latest, future-proof features and visions.

  • Plone - The original, open-source, enterprise-grade, all-in-one content management system written in Python.
  • Zope - The original Python web application server - the foundation for Plone and inspiration for Guillotina.
  • Volto - Plone's snappy, modern React front end powered by the RestAPI.
  • Guillotina - A re-imagined asynchronous back end compatible with Plone's RestAPI.
  • Pyramid - Pyramid is a small, fast, down-to-earth Python web framework.

With 295 attendees from 30+ countries from over the world, from Jamaica to Japan, USA to the UK, Germany to Finland, and Austria to Australia the conference was a definite success. We had newcomers to the community, as well as long time contributors since the early days of Plone. 

With the help of our sponsors, like iMio and Syslab, and many others, the conference was held 100% Online, using LoudSwarm platform by Six Feet Up. LoudSwarm nicely combined video streaming, recordings, schedules and chat discussion.

Every training, talk, and presentation was recorded and will be available online on YouTube later.

Group Photo 2020

Group Photo of Plone Confercence 2020.

Training

The conference contained, like in previous years, over 30 hours of free training, with topics including Mastering Plone, Volto Addons, Pyramid, and Guillotina.

https://2020.ploneconf.org/schedule

The training videos are already publicly available on YouTube.

Talks and Open Spaces

4 days of talks included almost 60 professional presentations, ranging from different use cases to building and managing projects into very technical talks about some aspect of a certain technology. Many of the talks focused on Volto, Plone's new React based frontend, and there was also lots of discussion about the future of Plone 6.

With Open spaces and 5 min lightning talks, it was possible to bring into focus many other aspects and topics.

Some highlights of talks: 

And many more! https://2020.ploneconf.org/schedule

Keynote Day 4

Day 4 Keynote.

Sprints

Every Plone Conference has contained 2 days of sprinting so after the conference talks people still gathered together and improved and developed Plone ecosystem forward.

Sprint topics included e.g.

  • Newbies Introduction to Plone Development
  • Plone Foundation Sponsorships
  • Plone Classic UI
  • ES6/Mockup/Patternslib
  • Volto
  • Unified Field for Images and Files (PLIP 2968)
  • Pylons Project projects (Pyramid, Deform, and others)
  • plone.importexport
  • Plone.org improvement

We will publish a separate sprint report on Plone.org to cover further details.

Community

Plone community aims to be the most welcoming and friendly community towards new and old people alike. We are happy to report that this conference had many new faces and we sincerely hope, that everyone felt the love.

On the communal aspects, check out the presentation Oh, the Places we've been! Plone 2001 - 2020

Every conference has a party, and this was no different, except beside the fact that everyone was online this time.

Thank you, everyone, sponsors, organizers, trainers, presenters, and especially participants for making this year's conference one to remember!

Community activity

Discussion during the conference.

Next year: Plone 20-year birthday, Plone Conference 2021 in Belgium!

This year's conference was in many ways very special, and next year even more so - Plone is celebrating its 20-year birthday, and if the situation allows, the conference will be held in Namur, Belgium!

So mark your calendars and stay tuned for Plone Conference 2021 news! Follow @ploneconf and #ploneconf2021 on Twitter and Instagram.

In Memory of Max Jakob

Posted by PLONE.ORG on December 12, 2020 02:01 PM

Every community has its silent heroes, people who contribute a lot to the community and put their heart and soul into it. In many open source projects, only the amount of code a person produces is what counts, but in the Plone community it is the person and each of their contributions that matters, be it enthusiasm, community work or the ability to motivate. Yesterday, the Plone community lost one of its silent heroes: Max Jakob.

Max was an unassuming yet integral part of the Plone community. After his first career, as a control systems electrical engineer in steel mills in remote parts of the world, he became an administrator at the Institute of Computer Science at Ludwig Maximilians University in Munich in the 1990s, turning to web technologies. Around 2000, he heard about Zope for the first time and co-founded the local Python and Zope User Group, which he hosted beginning in 2003. In the years that followed, he worked extensively with Zope and then Plone, using them for a wide variety of applications. He was the driving force behind the Munich and emerging German Plone and Python community.

Even though Max never directly made large code contributions to Plone or Zope, without him, Plone would not be what it is today. He created and fostered a vital and very active Plone community in Munich, extending to all the German-speaking countries. Many members of the Plone community were initially attracted to and eventually became integral to the community thanks to Max and the example he set. Two such members are former Plone Foundation board members Philip Bauer and Alexander Loechel, who were instrumental in bringing Plone to Python 3. Max was the rock without whom the Deutsche Zope User Group (DZUG) conferences in Munich, the PloneKonf 2012, and the Plone Tagung 2019 would not have been possible. He hosted many sprints and all of the World Plone Day events in Munich. Through his commitment and support, Max made possible the Munich Zope/Plone, Python and PyLadies groups. He was also a founding member and longtime board member of the German Python Software Association.

His application for Plone Foundation membership more than reflected his understatement. In it, he wrote that he would not have wanted to step into the limelight except that his friends had insisted he apply simply so there would be an official list of his contributions to Plone.

Max was a role model for the Plone spirit. He built a community for Plone and a better world led with trust, empathy and hope.

Max attended every Plone conference since 2007, in Naples. Plone community members from around the world have had the pleasure of meeting him at conferences, the Plone Open Gardens in Sorrento, or one of our many sprints. All of us who had the privilege of getting to know Max will carry him in our memory: a random meeting at a gas station in Brasilia, where Max, Alan Runyan (co-founder of Plone), and Alexander Loechel stayed chatting for hours; walking through the cobbled streets of Ferrara trying to talk Max into hosting a conference...and being surprised by a quiet smile and a “Yes”.

Max Jakob passed away on December 11, 2020. He fought an unwinnable fight against cancer and lost it during one of his favorite events, the Plone Conference 2020. The Plone community has lost a notable and beloved Plone ambassador and Foundation member. He will be cherished and honored by the community. May his spirit live on.

We raise a glass of his beloved IPA to wish him a good farewell. Rest In Peace, Max!

The Plone Foundation Board of Directors thanks him for his many services on behalf of the community and expresses its deepest condolences to his widow and family. 

max-jakob-collage-01.jpgmax-jakob-collage-02.jpgmax-jakob-collage-03.jpgmax-jakob-collage-04.jpg

Maik Derstappen: Plone 6 Theming with Diazo

Posted by Maurits van Rees on December 10, 2020 10:16 PM

How does Plone theming look in classic UI?

  • html5 theme plus a mapping configuration
  • deploying themes as ZIP-file for shared hosting is possible
  • With Diazo you can map any Plone html to a static theme layout.

Separating frontend and backend theme. Don't reinvent the backend views! You could theme the backend, so for content editors and admins, but it looks fine, not needed. You should focus on the frontend layout for visitors. To use the default backend layout you can include backend.xml in your rules, with some conditions.

Diazo is not for everything! If the backend markup differs from what you need, do not try to solve it with Diazo or XSLT. Instead, fix the backend templates directly, most likely with a z3c.jbot override.

You can use theme fragments or browser views to add new templates. The theme fragments can also be used as tiles in Mosaic.

I like using SASS mixins. Say you have a <div class="main-wrapper container">. Cleaner would be <div class="main-wrapper">. You can do this with mixins with @include in your selector.

For more information on Diazo, see https://diazo.org.

New features in Plone 6

From the backend we get Bootstrap 5 compatible html. Result is that Bootstrap themes are easier to integrate in Plone 6.

You have custom CSS in the theming control panel, for small changes. This actually sneaked into Plone 5.2.2 as well.

We have simplified Diazo rules.

Create a theme with plonecli:

  • pip install plonecli
  • plonecli create addon plonetheme.yourtheme
  • cd plonetheme.yourtheme
  • plonecli add theme
  • plonecli build

Theme from this presentation will be published as collective.bunapisicuto when it is ready for you to inspect.

Stefan Antonelli: Plone 6 Theming from Scratch

Posted by Maurits van Rees on December 10, 2020 05:08 PM

How to create a theme for Plone 6. Quite easy, because the templates use Bootstrap 5 classes. We build a theme from scratch, no Barceloneta, no Diazo.

First step is to create an empty plone_addon package with plonecli or mr.bob. For the questions you can answer: use Plone 5.2.1. We will switch later. My theme is plonetheme.munich.

I recommend to cleanup the standard package a bit. I remove tests, constraints for Plone 4 and 5. Check it out in the commits.

Now switch to extend Plone 6 and run the buildout.

You can add theme structure with a bob template, but I prefer creating my own.

Some interesting files:

  • package.json lists various tasks, especially the watch task.
  • In the theme manifest.cfg we more or less disable Diazo by emptying the rules line.
  • The compiled CSS and JavaScript are registered in registry.xml.

You can compile SASS to CSS using npm or yarn. Do yarn install in the top of your package. Later, with yarn dist you make it ready for production.

After these steps Plone is partially broken, or at least ugly. I do some basic fixes and it looks better.

For templates that you need to change you add z3c.jbot overrides. Personally I always kick out the "search in section" checkbox.

I don't like columns, but for this example I kept them. In most cases I need just one column. Plus maybe a side bar for portlets, but portlets must die.

With plonetheme.tokyo everything is Bootstrap, no columns, so no portlets, really fully responsive. This was the package where we built on Barceloneta Plone 5.2 and introduced lots of template overrides to put in Bootstrap. For Plone 6 we can just remove the overrides.

What about the toolbar? Yes, we dropped it. We bring editing features and navigation together. This is now a few feature: collective.sidebar. It is only one template to override. It works for Plone 5.2 at the moment, and I may work on it for Plone 6 during the sprints.

Question: is TTW still a viable path?

Answer: I like to concentrate on one path. I am not an expert in TTW theming. I switched to file system, except really small customizations. For small CSS customizations there is a field in the theming control panel.

Stefan Antonelli and Peter Holzer: Modernize Plone’s Classic UI

Posted by Maurits van Rees on December 10, 2020 04:02 PM

What was new in Plone 5? We had beautiful new theme: Barceloneta. Diazo theming by default. We switched to CSS compilation with less.

During the Tokyo conference Stefan thought up Tokyo Theme. Clean responsive theme for Plone 5. Tons of overrides to tackle problems in Plone 5. Issue with navigation and editing on mobile we solved with collective.sidebar.

We had community discussions, especially during several Plone events. Everyone tried to use Bootstrap (components). First idea: map variables from Barceloneta to Bootstrap, because they have similar ideas using different terms.

We have PLIP to modernize markup in templates, and another PLIP to modernize the default theme: Barceloneta LTS. Forms using z3c.form are already using the new classes.

Make things easier: UI, development. Creating a modern UI for the web is complex. You need to support different devices, responsiveness. In Bootstrap there are patterns for most useful things.

Developer perspective: expect one way to do things. Developers should not have to worry about design. When busy in the backend, you should focus on Python, not on it looking nice and shiny. Don't think about markup, just use components. The good news: there is documentation. The Bootstrap documentation is our documentation.

What is new in Plone 6?

  • Volto is the default UI.
  • There still is the Classic UI with Barceloneta look and feel, but updated.
  • No TTW (through the web) theming.
  • But there is a textarea to add some simple CSS (already in 5.2.3).
  • Some CSS variables may be changeable TTW.
  • Finally jQuery 3

Bootstrap is still the most popular front-end framework. Well documented, tested and maintained. It is so easy to create stuff, I enjoy it a lot.

What is new in Bootstrap 5?

  • Improved overall look and feel.
  • Updated and extended the color system
  • Custom properties: css variables
  • SVG Icon library
  • Pure javascript
  • Dropped IE10 and IE11 support
  • Bootstrap 5 is currently alpha 3.
  • See https://v5.getbootstrap.com

Features: what do we get from these changes?

  • Core templates use Bootstrap 5 markup. Instead of overrides in plonetheme.tokyo, we have lots of branches for the actual packages.
  • All major templates have been touched already.
  • For the current state, see the unofficial demo at https://classic.plone.de
  • The Bootstrap documentation has lots of snippets than you can copy.
  • You don't need much more CSS on top of it if you paste most examples. We added a little for own components, like navigation.

Barceloneta appearance is fully customizable. It is basically on opinionated set of bootstrap variables. Every aspect can be changed with variables: colors, fonts, sizes, spacings, grid gutters, etc. There are overall properties, like shadowed, rounded, gradients. Just turn on or off.

Theming workflow. plonetheme.barceloneta will also be published as npm package. bobtemplates.plone will have a template for the new theming workflow. You can do quick and dirty customizations through the CSS overrides field in the theme controlpanel.

Diazo will still be there, will work as before. Some optimizations in the rules.xml to make content are customizations easier.

How to deal with icons? What if you want to change the content type icons? Used to be hard. Now we come up with the idea of an icon resolver. We decided to use the bootstrap icons. Icons are registered via GenericSetup, for example with a record name plone.icon.alarm pointing to an SVG. You can then override this in your own GS profile.

Example icon use:

<tal:icon replace="structure python:icons.tag('love', tag_class='custom-class', tag_alt='foobar')" />

You get get back an inline SVG or an image tag.

Note: all z3c form widgets in Plone are now in plone.app.z3cform, and not scattered over lots of packages.

We will restart our weekly Plone 6 Classic UI sprints, starting Januari 13 2021, 10:00 (UTC+1).

Keynote: The User Experience - Editing Composite Pages in Plone 6 and Beyond

Posted by Maurits van Rees on December 10, 2020 02:55 PM

It may be a surprise to non-technical people to learn that pages created in Volto are not currently interoperable with traditional Plone's page editing. If you think about it, the reason becomes obvious. Volto, like Mosaic, creates tiled layouts, and like Mosaic it stores page data in special fields for the individual blocks and their layout. Neither Volto nor Mosaic pages are editable in TinyMCE, which expects just one rich text field.

Is this divergence between sites created in Volto and sites created in traditional Plone a problem? It does make it harder to describe what Plone is, and it might mean that there is no way to mix both approaches - for instance when part of a larger site is available as a Volto-based sub-site. Would it be possible to have one tool and one representation for tiled layouts so that we can avoid this divergence? Is there some other solution? Is it even a problem? Will Plone 6 be backwards compatible with Plone 5 and include a smooth upgrade path?

We will tackle these questions in this strategic panel discussion, moderated by Sally Kleinfeldt. Panelists will include Paul Roeland, Philip Bauer, Timo Stollenwerk, Victor Fernandez de Alba, and Eric Steele.

First, Philip has a message from Max Jacob who is very ill with pancreatic cancer and may not survive this weekend. He wants to thank the Plone community for what they allowed him to do: like organize the German Plone Konferenz. Thanks for all the friendships. Such a pity that this is happening now, he wanted to jump into only doing Plone for the next few years, due to changes at his job, and looked forward to that.

Classic, Mosaic and Volto Pages

We have Classic, Mosaic and Volto Pages. They have different internal representations and are not compatible. Is this a problem? Is there a solution? Is one tool, one representation possible? If we really need three, how to position?

Paul: for me as user this presents a problem. When do you switch over your site? We would like to not write 700 pages from scratch, again, like we did for previous composite pages.

Timo: We migrated quite a few large projects from classic Plone to Volto. One of those had collective.cover (other composite page system). Problem in general with such systems, is that they are pretty specific. They solve specific use cases and come from different eras. After any migration, it will not look the same any more. Whatever you do: the page will initially look ugly. So you put a lot of effort into migration, but then have to put manual effort into every page anyway. It can help: you at least have a start. We created a system where we migrated overview pages, and editors could click to migrate other pages one at a time.

Philip: We have code to migrate from non-folderish to folderish content types. There will be code to migrate to dexterity site root on Plone 6. We can make sure to migrate any standard content types. Mosaic is another story. So for pages you would at least have the text available. Maybe only visible for editors to pick and choose from. You may lose portlets, unless they get implemented in Volto.

Timo: When you go to Plone 6 and do a redesign at the same time, then you can jump on Volto. Otherwise you could stay at Classic Plone for now. There will be an overlap period.

Victor: For Mosaic you could dump all tiles into html and insert it in a block.

How to have a big Classic site with a subsite made in Volto?

Victor: Definitely doable, though we have not done this ourselves.

Sally: But what happens when an editor in Classic goes to the Volto subsite? The Volto page would not be editable then, right?

Philip: You should not offer this. I see no upside, no use case. Split them into separate applications, with shared authentication maybe.

Paul: Use case: large site with several departments. The marketing department may want snazzy new Volto things.

Timo: Just create another site then.

Cost and benefit of upgrading big Classic Site to Plone 6

So you just had a big migration to Plone5, and now what would you get for going to Plone 6.

Philip: We have this discussion every major upgrade. Communicate every upgrade as a relaunch. The relaunch is the reason for the upgrade. "There is a new version so you need to upgrade" does not fly for my clients.

Timo: We became a very developer oriented community, and every develop understands the need and benefits. We should really get back to giving more value at major releases, so clients really want to upgrade themselves. Plone releases should sell themselves.

Eric: It looks like we think Plone 6 + Volto is a costly upgrade with lots of benefits. For Classic 5 to Classic 6 the upgrade is not costly.

Alex Limi's vision for Deco: are we there yet?

Was this fulfilled by Mosaic? Volto? Something still needed?

Paul: Not quite, but slowly getting there. For me it would be Volto, plus some power features that Eau de Web (EEA) adds.

Timo: I think we went beyond what Limi envisioned.

Eric and Victor: What we have seen from Volto, is pretty close to what Alex wanted.

Victor: We gave the users powerful tools, so beware of them.

Philip: Partly yes. Volto is close, and it is for normal users.

Now some questions from users.

Migrations

Philip: Plone 4 to 5.2 was three migrations in one. Plone 6 is less of a problem.

Eric: 5.2 had a lot of backend migrations. A split between backend and frontend with plone.restapi in between makes things easier.

[The question on multiple variations of Volto, especially editors, went a bit too fast for me to write intelligible notes down.]

What's the future of using CT's/behaviors in Plone to design information architecture?

With Volto the trend seems mixing/adding 40 different blocks for every page.

Timo: Blocks are definitely the way to work. But the underlying power of content types and behaviors still exists.

Philip: We need blocks that represent a field or a behavior. That is unavoidable.

Next steps

Timo: We plan to have an open space on page compositions and Volto, and want to sprint on it.

Paul: Good if there is a longer term vision. I would rather have more power that a Site Admin can lock down, than having to choose between three different versions. I don't want choice stress.

Lightning Talks Wednesday

Posted by Maurits van Rees on December 09, 2020 06:21 PM

Lukas Guziel: Continuous Deployment

CD means deploying code automatically. It saves time, reduces human error. It Gitlab you can add gitlib-ci.yml and configure it. Include a base template that you use in multiple projects. End result can be a site that the customer can test.

Erico Andrei: World Plone Day 2021

We are a global community. Almost 300 people from 36 countries are at this online conference. World Plone Day is an annual Plone event. Next year of course online. April 28th 2021.

We want to stream 24 hours live on our YouTube channel. Showcase Plone. Technical talk, use cases, interviews, demo.

It should not all be in English, please use your own language. Talk to your local community.

Please help and join. See https://plone.org/events/wpd

Andreas Jung: collective.contentsync2

Syncing content between Plone sites through plone.restapi. It is a behavior. You have a source Plone site and one or more target Plone sites. You need Plone 5.2 under Python 3.

  • Create a dedicated user account with global role Editor.
  • Configure on the content sync control panel.
  • Automatically creates two content rules to sync content when added or modified.
  • You can enable it on all content types, also Folders.

See https://github.com/collective/collective.contentsync2

Philip Bauer: Why relations are weird

These packages have a part in relations:

  • zc.relation: abstract relation catalog
  • z3c.relationfield: fields and values on objects
  • plone.app.relationfield: converters from field to widget and vv
  • plone.app.z3cform: widgets
  • mockup: actual widget UI

In a schema use a RelationChoice field with vocabulary plone.app.vocabularies.Catalog, and set pattern directives.

It is not straightforward. So I wrote collective.relationhelpers.

See https://github.com/collective/collective.relationhelpers

Maybe use uuid instead of all this code.

[About ten people in the chat want to merge this package into plone.api. Actually, see this issue.]

Christopher Lozinski: Simple JSON Schema GUIs

Create a JSON schema, automatically generate the UI. Search for basic JSON editor library. He shows JSON in ZODB, so you can browse it, if I understood correctly.🙂

Eric Brehault: Second Guessing the Single-Page-App Pattern

Posted by Maurits van Rees on December 09, 2020 05:42 PM

SPA (Single Page App) is about providing an entire app by exposing a single physical web page containing an enormous javascript bundle. It breaks the original web paradigm in many ways. Surprisingly enough, we invest a lot of efforts to mimic the regular web behaviour.

Isn’t it time for modern frontend to reconsider the SPA approach?

[Note: Eric presented by using a projector to show his slides on a black Plone conference T-shirt. :-)]

Why are we doing this? Originally we always requested a whole page and this was considered slow. But we have good bandwidth now. And if you don't have good bandwidth, the super big bundle is not good either.

With SPA we try very hard to bring back the original working of the page, especially the browser history, being able to browse and then share the link to the current page.

To mitigate problems, we created an enormous stack. And we deny the complexity. New tools create new problems, even when their individual creators does not see the complexity.

"SPA isn't stable or efficient." But there is no way back. For example, you cannot create Google Docs with server side rendered pages. Web 2.0 is 15 years old. It is still about content.

SPA is separation of concern, which is a good principle. But we mix the browser layer (how you get and view the page) and the content layer (the page content).

It seems a take it or leave it situation: either use SPA or don't. What do we want? We want proportionate complexity. Do we need 100 percent SPA?

You can use micro components, see for example the demo of Maik Derstappen in the lightning talks on Monday, using Svelte. Micro frontend is bigger than that. It is a part of the application, that you develop separately. For example, you could do the Plone Sharing page like this.

Can we compile each page separately? Then each page is an app.

ES6 native support would be interesting. Combine with HTTP/2 and you need no bundles. Bundling is the most brutal thing ever. Horrible. Get rid of it.

Respect the layers. SPAs are monolithic. Break them down.

We should have a generic browser layer, common to many different use cases, for example for logging in. I don't want to code that, but plug it. Second step: push this layer to the browsers themselves.

Asko Soukka: Deploying Plone and Volto, the Hard Way

Posted by Maurits van Rees on December 09, 2020 04:22 PM

Here are the slides.

How about building Plone without buildout? Running Plone on Python 3 without WSGI? Deploying Plone and Volto with containers without Docker? Building all this in re-usable and safe manner in sandbox with restricted network access with Nix? Welcome to hear about our hipster setup where we lock, build and configure Plone deployments with Nix, insist to keep ZServer running on Python 3 for the love's sake, build software deployments into standalone tarball archives, and run them with Nomad – the simple on-premises-friendly alternative for K8S.

  • The easy, documented way: buildout, WSGI, Docker (if you need containers), Registry.
  • Our way: pip, TxZServer, Nomad, Nix

When you use a container infrastructure, you have multiple containers for running a Plone site, for example zeo clients, zeo servers, load balancer. Nomad helps there, and is much simpler than Kubernetes. We have one job file to rule them all: task groups, instance count, update policy, server resources, volumen mounts, tasks, consul services, vault secrets, environment variables, exec artifacts.

Nomad has "isolated fork / exec driver". No docker image needed. We have a Nix-built artifact, a tarball that we extract in the root of the container.

With Nix, you get 100 percent reproducible artifacts. Production equals development. You have a full dependency graph. The result is a standalone tarball, perhaps 100 MB. Disadvantage is that there are no conventions, no metadata, no shared layers, no documentation. It needs learning and practice. Well, some documentation now: https://nixos.org and https://nix.dev, partially made by people that were using Plone previously.

Some ugly parts from Nix:

  • Every language has their own Nix-conventions
  • dependency generator ecosystem is comples
  • cyclic dependencies are not supported
  • no storage device is big enough for /nix/store

Our (legacy) approach for Plone 5.2.1 without Buildout and with pip:

  • generated requirements.txt with buildout
  • create Python env with pip and nix
  • use pip-branch of z3c.autoinclude
  • disabled <includeDependencies />
  • generate instance skeleton with nix
  • forked plone.recipe.zope2instance

Plone 6 without Buildout should be pip-installable out of the box, but that is hear-say.

We use TxZServer in production, so ZServer using Twisted.

Nicola Zambello: Theming Volto without SemanticUI: Is It Possible?

Posted by Maurits van Rees on December 09, 2020 02:22 PM

We will walk through the process of building a product for Italian Public Administrations using a bootstrap-based theme. I'm presenting io-comune, RedTurtle's first product based on Volto and the strategies we used. We will see the possibilities in Volto for theming without SemanticUI, using bootstrap and sass and what are the next ideas we could work on.

Scenario:

  • We wanted to adopt Volto in our new project.
  • We needed to include Bootstrap.
  • Volto uses SemanticUI instead.
  • Two such frameworks will conflict, for example fighting over the same selector.

We tried. We tried harder. A cheap approach did not seem possible, so we looked for a sane one.

A new theme: pastanaga-cms-ui. Load only the CSS needed for Volto admin UI, see Volto PR 970. And public-ui for public pages. In your src/theme.js do not import the css/less from semantic-ui, but the pastanaga-cms-ui. In theme theme.config also use pastanaga-cms-ui. Also razzle-config.

You should normalize your base style, for example:

body.cms-ui {
  .public-ui {
    font-size: 18px;
  }
}

and wrap your components with .public-ui.

Building a product:

  • Base common package for every customer: https://github.com/RedTurtle/design-volto-theme
  • New intermediate layer for SemanticUI
  • New config layer for razzle/customizations
  • Template for actual projects: design-volto-kit, with a Yeoman generator: create-italia-volto-app

Erico Andrei: Building a Collaborative News Platform with Plone

Posted by Maurits van Rees on December 09, 2020 10:20 AM

Here are the slides.

With some other people I have created pendect.com. The idea was to build a TL;DR (Too Long, Did not Read) platform. Create short cards, linking to long articles. Cards are submitted by our community (Pentributors). Also: for every card, we plant a tree.

Technical requirements:

  • Collaborative workflow
  • Permission control, who can do what, where and when
  • Metadata, categorization
  • We are a startup, so we need SEO.
  • Open source

We decided for Plone as Content Management System. It has most features out of the box. I know Plone of course. But I considered building a simple API on top of Pyramid. But with Plone we could go to market immediately. It has a proven record with news portals, and also a friendly community with smart people.

We use DBpedia for Metadata and categorization. This works on a Wikipedia dataset. They have a tool Spotlight to detect entities in texts. We use a Sparql connection to DBpedia to query for more information.

We use a bit of everything else: Cloudflare, nginx, varnish, HAProxy, Ansible, Thumbor, Sentry, Mailhun, IFTTT, Zapier, Gravatar. We already survived twelve different surges in traffic from Reddit. We have tens of thousands of users. We use Thumbor for image scales. Actually easy to integrate in Plone.

Our toolset and add-ons:

  • We have development speed thanks to some friends, like Python 3 with F-strings, Type hints (I put them in from the start, helping me to think about my code), data classes (better than returning dictionaries everywhere). Also: black, isort, flake8, PyCharm.
  • We do not use many Plone add-ons: collective.z3cform.datagridfield for table-like content, collective.sentry to talk with Sentry, contentrules.slack to send messages to Slack, souper.plone to store many small data records/

Content types:

  • Default folders, documents, images, collections
  • Category: collection behavior with sub-objects
  • Card: similar to news item, but more categories

Adapting Plone:

  • Browser views: dashboard became "My Feed", Author page became the profile
  • Some content rules

New features:

  • Aggregator pages for tags, people, locations, organizations.
  • Development with souper.plone: you can follow categories, tags, people, locations, organizations, and Pentributors

Sync to archive.org. Takes too long, so this needed to be asynchronous. Also other external services, like translating.

Lessons learned:

  • Always create upgrade steps. Be aware of existing registry configurations: use purge=false.
  • Plone training materials are the de factory documentation for Plone.
  • I hate the current resource registries.
  • Plone lacks a simple and working async/delayed solution.
  • Webp images are not for all. Be careful with caching them.

Future:

  • Translation and auto summary to all Pentributors.
  • Card threads, similar to Twitter threads, using content relations.
  • Use ElasticSearch.
  • Move to RelStorage.
  • Plone as headless CMS, with Volto and mobile applications
  • Move away from the default user folder, it is getting slower already.

Lightning talks Monday

Posted by Maurits van Rees on December 09, 2020 10:19 AM

Alec Mitchell: WYSIWYG problems be gone

A new add-on to vastly* improve your content editing experience.

(* size of improvement may vary, no warranty implied. The following is (un)paid free software promotional content)

Adding images to a document is so hard! At least nine steps! (Difficulty be exaggerated for marketing purposes). Why can't you drop an image in? You can, with our new, super special add-on krcw.tinymce_imagedrop.

Can you drop two? Yes!

Can you drop more than two? No, because browsers are weird.

But it fails gracefully.

Steve Piercy: Deform and friends

How I learned to stop worrying and love web forms. We must have a good interface, data structure, validation, security. Deform (form library), colander (de/serialization), peppercorn (data structure), bootstrap forms for design. We have a looooong list of widgets. In deform 3.0 we will use bootstrap 5. See https://github.com/pylons/deform

Christopher Lozinski: Forest Wiki

The Forest Wiki is a modern version of Zope 2. Biggest difference: it uses Pyramid's security and views. Modern JavaScript enabled ZMI: reorder, sort, rename, etc. Both WYSIWYG and MarkDown pages. Advanced types like JSON, CoffeeScript, pug. Pug is the leading template engine for Node.

Jens Klein: RelStorage

Plone relational database backend storage. It is a drop-in replacement for FileStorage of ZEO. You can use PostgreSQL, MySQL, Oracle, SQLite. It has been around for about 13 years, grown old, but in recent years development has picked up, driven by Jason Madden, including Python 3 support. It is much more performant. Latest release 3.4.0 is form October 2020.

PostgreSQL is the cloud database, kind of industry standard, well supported by all big cloud providers. Easy to install in Docker.

Advantages of RelStorage: fast, parallel commits, better concurrency, shorter locks. Optimized per process caching. Blobs in database. Optionally you can use it in history free mode. You lose the Undo functionality, but you don't need to pack so often.

plone.recipe.zope2instance supports it with the rel-storage option.

You can use additional client side caches, shared between all threads of a process.

With the zodbconvert tool you convert from ZEO to RelStorage, or the other way around, including converting blobs if needed.

ZODB keeps old transactions, so packing is needed, even in history free mode. RelStorage has a fast zodbpack.

Blobs:

  • RelStorage 3.x is Python 3 only and runs with Plone 5.2+. Here, blobs should be stored in RelStorage.

  • RelStorage 2.x is for Plone 5.0, 5.1, and blobs should **not**  be stored in the database, except for Oracle backends, otherwise you should still use a shared blobs filesystem directory.

    System Message: WARNING/2 (<string>, line 69); backlink

    Inline strong start-string without end-string.

I use RelStorage today for all my live deployments. I have used it since version 1.6 with Plone 4.3 and never had problems. Always blazing fast. Dev/ops and sysadmins love it: it is a standard solution, nothing special, just works.

Maik Derstappen: Add-on catalog for Plone

We want to bring back an add-on catalog for Plone. You can look on PyPI, but it is hard to find packages.

We worked on a tool for this. You can search on named, filter on Plone versions and add-on types.

Components:

We only aggregate packages that have classifier Plone :: Framework. We will probably work on this during the sprints and are happy to onboard you.

David Bain: Plone and Webflow

Both platforms are for building websites, but they approach things in different ways. I hope this may inspire. Keep in mind the motivation of the two platforms, which may account for some strengths and weaknesses.

Webflow is visual web design, less content management. Strong design tools. Designer friendly layout tools. You can design a page with what you could call blocks.

Plone is enterprise content management, focus on security. Linking to an attachment is standard, where it is tricky in Webflow. Forms are way more flexible.

We have also built a website in Webflow and based it on Plone.

Miu Razvan: Volto grid block

  • Created by Eau de Web team
  • Dependencies: Volto blocks form
  • Similar component: Volto columns block
  • Use it to organize other blocks.
  • Demo showing lots of configuration options, including for different screen sizes
  • See https://github.com/eea/volto-grid-block

Maik Derstappen: Custom elements

Custom elements are an extension to normal native html elements, for example <flag-icon>.

The promise of web components: write once, use anywhere. See https://custom-elements-everywhere.com/

How do you use this in Plone? Use plonecli add svelte_app to create a small app. Run yarn. Install in Plone add-ons control panel. Edit a page. Replace html source with <my-svelte-app />. And your component is there and working. The size is less than five  kilobytes.

Tiberiu Ichim: volto-slate

volto-slate is a drop-in replacement for the standard rich text editor in Volto. Volto turns an HTML document into a modern document.

Why another text editor instead of improving the existing one?

  • With Slate we get a better plugin framework. Plugins are just wrappers around the editor. The standard Draft.js is meant to be integrated directly by an application, no concept of plugins out-of-the-box.
  • Slate has simple DOM-like storage for its values, making it easier to render the ersult.

Current status:

  • No migrations of any kind.
  • Right now not possible to completely remove or replace Draftjs out of Volto.

Alin Voinea: Volto Dexterity Schema and Layout Editor

Posted by Maurits van Rees on December 08, 2020 05:40 PM

Through the Web Dexterity Content-Types with Schema Editor and Blocks Layout Editor

How do we define content types schemas in Plone?

  • TTW schema editor
  • GenericSetup profile
  • Behaviors, schemas in Python

Why do we need them, we have Volto blocks, right? You still need metadata, a title, etcetera. Certainly for larger institutions you need a structure, a schema. Volto itself has schema-based components.

Layout editor. Blocks have properties, like a placeholder, a position. You can type text in a block: "Published on date by author". Then select "date" and link this text to the published date metadata, and select "author" and link it to the author. Save this as a layout for a content type. You can export this to a JSONField in a custom behavior, so you can save it in version control for production.

List of add-ons and other packages that make Volto awesome: https://github.com/collective/awesome-volto

Jens Klein: Performance, Profiling, Power-Consumption

Posted by Maurits van Rees on December 08, 2020 04:27 PM

I want to focus on Python performance, so not caching or database performance.

Tools:

  • py-spy: Overall mix of the whole live application, top-like.
  • repoze.profile: WSGI middleware, slows down application. Profile single request and analyse its call stack by count, call time, etc.
  • dis: disassembler for Python at the bytecode level.

Improvements Plone 5.2.0-5.2.3:

  • Avoided early providedBy calls
  • __getattr__ early exit on common attributes
  • zope.interface: some functions are called hundreds of thousands of times when you reindex an index, so a tiny improvement helps a lot. I found various places that could use improvements, and that landed in the package, together with memory improvements by Jason Madden.

Live demo. I call py-spy with sudo because I need to connect to an existing process id.

Future Todo's:

  • plone.restapi has optimization potential, all navigation related, but currently it still supports even Plone 4.3. This will likely wait for a 5.2-only or Python3-only branch.
  • plone.registry is called too often
  • Use python: expressions in all page templates. They are way faster than standard Tales expressions.
  • More introspection.
  • Move more logic from page templates to Python code

Advice: start introspecting the performance of your application.

Alex Clark: The State of Pillow

Posted by Maurits van Rees on December 08, 2020 04:26 PM

The Plone Conference account tweeted that a State of Plone talk would be awesome and that the Plone community missed me. I miss the Plone community too, so I am here.

I will state it clearly: Pillow would not exist if not for Plone.

In July 2010 I announced Pillow as "friendly" fork of PIL. The mailing thread and future answers are interesting to read.

Some history:

  • 1991: Python 0.9.1
  • 1995: PIL started
  • 1998: Zope
  • 1999: Zope2
  • 2000: Python 2.0 with distutils
  • 2001: Plone
  • 2005: Buildout
  • 2006: I attended my first Plone Conference, in Washington
  • 2006: setuptools was born

PIL had an issue, or Plone had an issue with PIL:

  • PIL used distutils.
  • Plone 3.2 used Buildout and setuptools
  • PIL was not installable in Buildout and setuptools
  • Specific problem: import Image could mean the Image module from PIL, or the Image module from Zope.

Various ways of repackaging PIL started, for example PILwoTk. You can still find various PIL derivatives at https://dist.plone.org/thirdparty/

PIL 1.1.6 from 2006 is still the last version on PyPI. I got maintainership of this page this year, actually. Pillow 1.0 is basically the same, except that it uses setuptools. This worked in buildout. I was happy.

Couple years, nothing really interesting happened. But some contributors came along. Pillow 2.0.0 in March 2013 had Python 3 support.

An important milestone in 2015: we added release notes.

Release schedule: in the beginning of every quarter.

We get some money from Tidelift for maintenance.

Fred van Dijk: collective.collectionfilter as a Light-weight Faceted Navigation or a 'compare' Console

Posted by Maurits van Rees on December 07, 2020 09:10 PM

I want to talk about some categorisation and classification options in Plone, next to the folder structure.

Faceted navigation: drill down on 'facets' when you search for items. It was popularized by online shopping. Facets in Plone for developers is: whatever is in the ZCatalog, and for users: what you can search on in Collections. Gold standard is eea.facetednavigation, developed for the European Environment Agency. Examples: EEA, and on two sites by Zest: Vaquums and Minaraad, where it replaces the standard search.

collective.collectionfilter is a much leaner, meaner, but also more limited version of faceted navigation. Demo with standard Plone News Items with some tags (also known as categories, also known as Subject). Add a Collection that filters on News Items. Now add collection filter portlets.

eea.facetednavigation takes over your complete page. In an action you enable or disable it.

Now a demo of collectionfilter in SGBP, a documentation website for water management planning in Belgium/Flanders. The customer wanted to take some graphs and compare them. We did that with collectionfilter and collective.classifiers. With the last one we added structured categories: one for water basins and one for parameters of the graphs. Now we use collectionfilter to query a parameter and show the graphs for all water basins.

You can adapt several things in the collectionfilter UI, for example change how search options are displayed. This is documented, but took me a while to get right.

Collectionfilter also works with Mosaic, because the portlets are also mapped to tiles.

Alexander Loechel: The Plone is Dead, Long Live the Plone!

Posted by Maurits van Rees on December 07, 2020 06:01 PM

Here are the slides.

A talk about the essence of Plone. What does Plone mean for somebody?

Plone is an awesome combination of vision, software and community. But what defines Plone, what is the essence, and how does it change over time.

At the Plone Conference 2019 I had the feeling of realizing why we sometimes speak of very different things and why the Plone of today is not the Plone we know from the years.

This talk wants to summarize this feeling of realizing, explain why we may have a problem of misconception. It is a mix of technological questions, the overall vision, branding and community topics in depth.

Plone is a software, an api, a community, a foundation. Plone core is an API, but there are more than one: plone.api, plone.restapi, the Plone UI, none is complete. Every Plone company has a different point of view of what Plone is or should be.

Plone is a mature open source Python CMS. As said earlier today, implementations change, but values do not. We want to simplify Plone.

What is or was Plone for me? The CMS product, the framework/toolkit, the vision, the community, the foundation. Plone is an umbrella for a lot of frameworks.

Anish Kapoor: "All ideas grow out of other ideas."

Plone was the User Interface that gave users the power of Zope and CMF. It is a layered system. Layers hide complexity, make complex things easier, you have a defined API between the layers. It should be possible to switch out one layer for another. For  example we switched out the Archetypes layer for the Dexterity layer.

We are the Plone community, not the Zope community. Values and ideas are more important to us than technology.

Problem for API: there are always undocumented features without API definition.

At Plone conference 2013, Paul Everitt showed a lesson from Zope 3: rename the beast. Zope 3 was very different from Zope 2, so it should have been renamed.

The vision lives on. Volto is the future of Plone UI. It is the essence of the Plone vision.

I want to thank all the active Plone community members.

The Plone Foundation has a mission: protect and promote Plone. Lots of decision making, but no coding.

Wishes for the future:

  • Endorse Volto, the Plone vision of simplifying CMS work
  • Attract new developers, keep the community vital
  • There is overlap with the Pyramid and Pylons family, we could absorb them.
  • Stay commented and learn from each other. There are no rock stars, everyone started small and is still learning from each other.

Asko Soukka: Plone and Volto in a Jamstack project

Posted by Maurits van Rees on December 07, 2020 05:54 PM

Here are the slides.

I am a software architect at University of Jyväskylä. I have been using Plone since 2004 and GatsbyJS since 2018. The university wanted one student information management system to rule them all, but... every organisation shall do their own integrations, using granular REST API with deep JSON responses. And there should be branded study guides, which we crafted with GatsbyJS. But this was not enough for the Open University part. They really needed a CMS.

We use Plone 5.2, Volto, GatsbyJS, and have 6000 html pages, times two languages, out of which 760 are Volto pages. With Plone we could extend content types without needing to do any coding, in the content types field editor. In volto we added auto-complete widgets with custom vocabularies. On the GatsbyJs side, we query the connected pages with GraphQL. We render Volto layouts with React components, rendering individual blocks.

Why did we choose GatsbyJs? It is a ReactJS-based site generator. Being static, it is very fast. You can use multiple sources as input, using a plugin architecture. Data lookup is done with GraphQL. It is easy to get started, with comprehensive documentation.

I mentored two Google Summer of Code projects for the gatsby-source-plone plugin. It supports default types and most TTW types, also Volto blocks. You can do incremental updates by modification date, so it is really fast.

Not everything is easy. The full "GatsbyJs experience" requires practice. You want to replace inline images and links with GatsbyJs images and links, replace file links with direct downloads.

Using @plone/volto as dependency to render blocks seemed like a good idea, but it required webpack overrides to be impartable, and could not be used for images and links.

The ugly parts of GatsbyJs:

  • The GraphQL source plugin cannot cache.
  • The build may take hours, and gigabytes of memory.
  • The build result in readonly.
  • For me it is hard to follow GatsbyJs development, especially individual plugins, because they use a monorepo.

Editors can work on the site during the day, and then wee rebuild the result during the night

Timo Stollenwerk and Victor Fernandez de Alba: Volto: Past, Present and Future

Posted by Maurits van Rees on December 07, 2020 03:46 PM

Most important milestone for Volto this year: we went from a small number of people to a critical mass of developers who work on Volto.

Plone has evolved from a product to a contract. Implementations may change, but values do not. Tiberiu Ichim: "Through the freedom that they provide,Volto blocks are a foundation for innovation that enables Plone to step in line with the latest state of the art for web development."

Volto is not only UI and UX for Plone, but it is a framework. We have design principles, like approachability, do not overengineer things, frontend and backend are meant to be decoupled, use semantic versioning.

Where are we in Volto today? 203 releases, 58 contributors, closed 1253 pull requests, 3024 commits. In which areas is Plone 6 better than Plone 5 today? Blocks engine, authoring experience, we have a schema and layout editor, the listing block, direct access to the full JavaScript and React ecosystem, performance. Bonus: Volto still works with Plone 4 as API backend. So if a customer is not ready to jump to Plone 5 or 6, you can give them a new UI on top of Plone 4. Every last Friday of the month we have the Volto Early Adopters Meeting. It is open for everyone interested.

We call Volto 4.0.0, from March 2020, the maturity release. And in nine months we went to Volto 10. We have huge improvements in extensibility. A new generator based on Yeoman will replace create-volto-app.

  • There is a new add-on architecture, using the mrs.developer tool.
  • API Block Transforms are great. You can serialize and deserialize input and output. We use this for example to handle resolveuid links.
  • Lazy loading
  • You can force a layout in the blocks editor, so Editors have to stick to this layout.
  • You can copy and paste blocks now, even on a different page in a different window.
  • Listing blocks: we talk about adding support for variations, removal of the listing container object from the results.

Future:

  • We want to bring back slots in Volto, a mix between viewlets and portlets, evolution of the Plone and Mosaic tiles concept.
  • Adoption of the Slate editor.
  • Albert Casado worked in UI improvements for us.

Plone Conference 2020 Off to a Great Start

Posted by PLONE.ORG on December 06, 2020 04:54 PM

Plone Conference 2020 Online started with two days of training sessions, gathering about 100 participants each day to 9 different sessions!

Training included topics such as:

  • Getting Started with Plone
  • Mastering Plone 6 (2 parts)
  • Volto Addons (2 parts)
  • React and Volto (2 parts)
  • Pyramid
  • Guillotina

Feedback from the training:

"Learned some exciting things about how to develop new addons for Volto/Plone 6. What a great first day at the virtual Plone Conference 2020."

"I'm really impressed about how easy it is to customize a new #Plone site using #Volto."
"I've done for Pyramid training, #ploneconf2020 's first program! It was the first time for my English conference. I could have a conversation because everyone was so kind. Many thanks, @steve_piercy and all attendee! I'm looking forward to Monday's keynote."

The training sessions were recorded and will be available to all Plone Conference participants.

Thank you trainers and thank you participants, see you soon again at the conference talks!

https://2020.ploneconf.org/

How to Participate: Plone Conference 2020 Online

Posted by PLONE.ORG on December 01, 2020 05:41 PM

All this information and more is available to Plone Conference attendees, so please make sure you get your ticket.

What is the conference language?

The Plone Conference is a global event with attendees connecting from a wide variety of countries. English is the official language, but Track 3 will be in French on Day 2.

What is the conference time zone?

The event’s official time zone is Central European, but LoudSwarm will show the schedule in your local time zone.

What technologies do I need to log in to for the conference?

LoudSwarm is the only requirement to be able to watch the talks, whether you are watching live or catching up later.

All conference attendees will get invitation link to LoudSwarm platform, and get access to all information and guides there.

But part of the fun of a conference is being able to talk to your fellow attendees and speakers. For that, we will use Jitsi and Slack.

Jitsi will be used for face-to-face interactions after each talk. Each speaker will join Jitsi after their talk to be available to answer questions or have discussions with the attendees.

Slack is our primary method of text-based communication. If you did not already receive an invitation, email [email protected]

How do I ask questions and network with other attendees?

A face-to-face Jitsi room is available for each talk. You can join this room during and after each session. The speaker will join once they are done with their talk to be available to answer questions.

The primary text-based discussion platform is Slack.

Do I have to turn my webcam or camera on?

Be ready to be on camera, it’s easier to connect with the other attendees when you can see each other face to face. Also be sure to check your microphone as you'll have the opportunity to ask questions to speakers, sponsors, organizers, and attendees.

Why is there extra time before the sessions start?

Come and join us an hour before the first talk each morning to get comfortable with the event and where everything is. It’s also a great opportunity to chat with others and network with other members of the community.

How do I ask questions?

In the LoudSwarm session, you will see a Sli.do widget on the right to ask questions to the speaker during the talk, and you can also ask questions in the Slack channel to continue the conversation with fellow attendees.

Will the talks be recorded?

Yes, all of the talks (including trainings) will be recorded and will be available to watch in LoudSwarm once they have been processed (usually about 10 minutes after the talk has finished). Recordings will be made available to the public one month after the conference.

I need help

Join the #conf2020_help_desk channel in Slack to ask the moderators your questions.

More information

All the needed information is delivered to attendees and can be found in LoudSwarm.

Get your tickets and get ready for Plone Conference 2020!

Plone migrations: importing collective.jsonify export data into ArangoDB (Part 3)

Posted by Andreas Jung/ZOPYX on November 27, 2020 08:47 AM
Plone migrations: importing collective.jsonify export data into ArangoDB (Part 3)

In the last blog post, I described how to export a Plone site using collective.jsonify to the filesystem as directory of JSON data. This blog post will explain briefly to import the JSON data into ArangoDB as interim migration database

Why ArangoDB? We are using ArangoDB as a document database for storing the JSON data. This allows us to introspect and query the data to be migrated and imported. This is often necessary in complex migration with unknown data and unknown object relations. Using grep or similar tools on a huge JSON dump is unlikely the right approach. Using ArangoDB also allows us to run partial migrations for testing purposes (e.g. for testing the migration on a particular folder or a particular content type).

ArangoDB is multi-model database (key-value store, document store, graph database) and is available as community edition, an enterprise edition and as cloud edition (ArangoDB Oasis). The community edition is sufficient for migration projects. There are various installations options for all operating systems, all Linux distributions and through Docker.

After the installation of ArangoDB, you need to create a dedicated user account within ArangoDB and a dedicated database. This can be easily achieved through the ArangoDB web UI.  We assume for the following steps that you have created a database named plone and an ArangoDB user account plone with password secret.

Preparing your target Plone site

We assume that the target Plone site is running on Plone 5.2 and Python 3. The core functionality is integrated in the collective.plone5migration add-on for Plone. Prepare your buildout like this:

[buildout]
extends = buildout.cfg

auto-checkout +=
    collective.plone5migration     

sources = sources

[sources]
collective.plone5migration = git [email protected]:collective/collective.plone5migration.git


[instance]
eggs +=
    collective.plone5migration

Importing your JSON data into ArangoDB

After running buildout with the configuration given above, you will find this generated import script:

 bin/import-jsondump-into-arangodb --help
usage: import-jsondump-into-arangodb [-h] [-d DATABASE] [-c COLLECTION] [-url CONNECTION_URL] [-u USERNAME] [-p PASSWORD] [-i IMPORT_DIRECTORY] [-x]

optional arguments:
  -h, --help            show this help message and exit
  -d DATABASE, --database DATABASE
                        ArangoDB database
  -c COLLECTION, --collection COLLECTION
                        ArangoDB collection
  -url CONNECTION_URL, --connection-url CONNECTION_URL
                        ArangoDB connection URL
  -u USERNAME, --username USERNAME
                        ArangoDB username
  -p PASSWORD, --password PASSWORD
                        ArangoDB password
  -i IMPORT_DIRECTORY, --import-directory IMPORT_DIRECTORY
                        Import directory with JSON files
  -x, --drop-collection
                        Drop collection

In the former blog post we created a JSON export in I/tmp/content_test_2020-11-25-15-40-59.

You can import this export directory into ArangoDB using:

bin/import-jsondump-into-arangodb -i /tmp/content_test_2020-11-25-15-40-59 -x -u plone -p secret -d plone

Output:

connection=http://localhost:8529
username=root
database=ugent
collection=import
import directory=/tmp/content_test_2020-11-25-15-40-59
truncating existing collection
truncating existing collection...DONE
......

A nice progressbar will should you the progress of the import operation. The import speed depends on your local system. A typical import of 100.000 JSON files (100.000 Plone objects) with a total size of 50 GB takes about 45 to 60 minutes (largely dependent on the IO speed of your disk).

collective.plone5migration has been developed by Andreas Jung as part of a customer project with the University Ghent (migration of the ugent.be site to Plone 5).

Plone migrations: exporting a Plone site using collective.jsonify (Part 2)

Posted by Andreas Jung/ZOPYX on November 25, 2020 03:17 PM
Plone migrations: exporting a Plone site using collective.jsonify (Part 2)

In this blog post, I describe to how to use collective.jsonify for exporting a Plone to JSON.

Buildout preparation

  • install collective.json (we are using our ugent-fixes branch. This branch will be merged back into the master branch soon)
[buildout]
sources = sources
auto-checkout =  
   collective.jsonify

[instance]

eggs +=
   collective.jsonify

[sources]
collective.jsonify = git [email protected]:collective/collective.jsonify.git branch=ugent-fixes  

Installation of collective.jsonify inside your Zope/Plone installation

bin/instance run src/collective.jsonify/collective/jsonify/scripts/install_migration.py

Run the migration export

bin/instance run src/collective.jsonify/collective/jsonify/scripts/migration_export.py <plone site id>

You will see some output similar to

2020-11-25 15:40:59 WARNING collective.jsonify export >> Directory /tmp/content_test_2020-11-25-15-40-59

Depending on the site of your Plone site and the number of Plone object, the export may take a while. An export on a larger customer site with 100.000 objects and 60 GB data, an export took about 90 minutes...these numbers may vary depending on disk and CPU speed. The export will generate an export directory in /tmp (by default). The directory name is based on the Plone site ID and a timestamp. You can specify a different export directory by setting the $EXPORT_DIR directory.

The next blog post will cover the aspect of setting up ArangoDB as migration data and importing the exported JSON data into ArangoDB.

Plone Migration approaches (Part 1)

Posted by Andreas Jung/ZOPYX on November 24, 2020 01:07 PM
Plone Migration approaches (Part 1)

This is the first of a series of blog posts that deals with Plone migrations. I am trying to tell the story about how we are doing large-scale Plone migrations which usually involves migration of code, switching add-ons and  reorganizing content. This is usually accomplished in our migration through an export of the original Plone site and "magic" importer script on the target system. This migration approach evolved over the years. Our current approach is based on the experience and the work over the last year when we worked on a large-scale migration for the University of Ghent (90.000 content objects, 50 GB of data) and some sites for an industrial customer in Stuttgart.

Migration approaches

In-place Plone migration

Plone 5.2 supports a so-called “in-place” migration which is usually the right choice for simple sites with simple content-types and standard add-ons like PloneFormGen. The database file of Plone 4.3 is migrated in-place and transformed into a Plone 5.2 compliant database file. This is the approach in theory. It is often necessary to perform additional work and additional migration steps to migrate an old Plone database to Python 3 format - in particular when the Plone site has a certain age and history (which often implies a lot of cruft and old configurations that can break or interfere with the standard Plone migration). We use this migration approach only for small and simple sites.

Export/Import using Transmogrifier

The export/import approach allows us to perform a clean export of the content from the old site and to perform a clean import into a fresh Plone 5.2 site. So the new site will not contain any cruft content or obsolete configurations etc. The export of the is usually accomplished using the collective.jsonify add-on for Plone which basically exports all content objects and their metadata as JSON - one JSON file per content object. We also have supplementary code to export additional Plone settings or e.g. the Plone users and groups (if need, unless users and groups are maintained in LDAP or Active Directory or something similar). The exported JSON files are the foundation for a fresh import using the Plone Transmogrifier tool. Transmogrifier is basically a processing pipeline for piping the JSON data into Plone. The import can be customized using so-called “blueprints” in order to inject project-dependent dependencies, filtering steps, etc.

Export/Import using a custom migration pipeline

In our larger Plone migration projects, we often use a variation of the Transmogrifier approach. The export of the data and content from the original site is also done using the collective.jsonify add-on. The exported JSON files are then being imported into a migration database (we use ArangoDB). The migration database allows us to perform a partial import, query, and inspect the migration data in order to work on migration aspects and problems more easily than searching to Gigabyte of JSON files on the filesystem with operating system related tools like “grep”). The import is done using a custom migration package (which is to 80% generic for most Plone migrations, the remaining 20% are project-specific adjustments or additions because every migration is different). Our migration package allows us to run partial migration in order to test migrations more flexible. E.g. the migration package allows us to import a subset of content (either one or more folder or only specific content-types). This is very handy and time-saving. Migrating a large Plone site with a lot of content can take many hours. So testing changes on a particular feature with a subset of the data in advance is always better than having a failure on the full migration set after some hours.

Some new Plone add-on releases

Posted by Andreas Jung/ZOPYX on November 23, 2020 04:05 PM
Some new Plone add-on releases

Today, we released some small Plone add-ons as we developed and used over the last years for customer projects. In many cases we duplicated the same over and over again...so it was time for some refactoring and moving the code into dedicated packages.

zopyx.plone.persistentlogger

zopyx.plone.persistentlogger supports persistent logging where the log data is stored on an arbitrary persistent Plone object (as annotation). Typical usecases are application specific logging e.g. for logging a history per content object directly in Plone rather then having a huge common log on the filesystem. The log entries are stored using object annotations.

zopyx.plone.persistentlogger
Persistent logging for Plone objects
Some new Plone add-on releasesPyPI
Some new Plone add-on releases

collective.metadataaudit

collective.metaaaudit logs metadata changes to Plone content objects transparently (for all content types). This add-on in in particular useful when you have to answer the question “who was doing what and when”.

collective.metadataaudit
Audit metadata changes
Some new Plone add-on releasesPyPI
Some new Plone add-on releases

collective.debug

This add-on provides a @@debug browser view that returns some debugging information on the current context object, the current user and its roles, the instance dict of the current context object and request information.

collective.debug
A Plone debug view for introspecting the current context, user, roles and instance dict
Some new Plone add-on releasesPyPI
Some new Plone add-on releases

collective.contentsync2

Content sync between Plone sites over plone.restapi

Features

  • sync content folders or individual content objects to a remote Plone site
  • full sync for initial sync
  • incremental sync for content updates
  • configurable sync behavior per content type
  • trigger immediate sync upon create or update operations through content rules
collective.contentsync2
Content sync between Plone sites
Some new Plone add-on releasesPyPI
Some new Plone add-on releases

Foundation Board of Directors Nominations Open

Posted by PLONE.ORG on November 20, 2020 10:56 AM

The Plone Foundation is a not-for-profit, public-benefit corporation with the mission to promote and protect Plone. It protects the trademark, copyrights and other intellectual property, hires the release manager, works with our various communities and committees and develops policies where needed. It is a rewarding and much appreciated way to give back to the Plone community.

The official announcement describes the nomination process and gives more information about the work involved in being a board member.

Please Consider Serving!

Plone 5.2.3, Plone 5.1.7 and Plone 4.3.20 released!

Posted by PLONE.ORG on November 20, 2020 08:15 AM

General notes:

Plone 5.2.3

Plone 5.2.3 is a bug fix release of Plone 5.2.

Download Plone 5.2.3

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.3/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Some highlights of this release are:

  • zope.interface: Fixed potential memory leak, see https://github.com/zopefoundation/zope.interface/issues/216. Fixed inconsistent resolution orders, see https://github.com/zopefoundation/zope.interface/issues/199.
  • Zope: fixes for a few template syntax errors. HTTP header encoding support.
  • A few possible information disclosure problems in handling of XML and of ical urls were reported by MisakiKata. They have been fixed by the Plone Security Team. Since they require an attacker to already have Manager or Site Administrator rights, we decided it was not necessary to create a hotfix for this. See https://github.com/plone/Products.CMFPlone/issues/3209
  • plone.recipe.zope2instance: added options clear-untrusted-proxy-headers and max-request-body-size.
  • Products.MailHost: support messages with explicit Content-Transfer-Encoding 8bit, see https://github.com/zopefoundation/Products.MailHost/issues/30. Note: in add-ons this may require changes to the tests.
  • mockup: fix plone toolbar action links being updated only on the first navigation action in the folder_contents structure pattern.
  • plone.staticresources: updated Bootstrap Icons to 1.0.0 final.
  • plone.app.contenttypes: allow passing a custom catalog-query to migrateCustomAT to constrain which objects to migrate.
  • plone.dexterity: make sure that Dynamic schema is updated on all ZEO clients on change.
  • z3c.form/plone.app.z3cform: fixed compatibility with changed repeat syntax in Zope 4.4, see https://github.com/zopefoundation/z3c.form/issues/94.
  • Products.ATContentTypes: drop use of test() in templates, unsupported since Zope 4.4.
  • Lots of deprecation warnings fixed, especially during startup.

For detailed changelog, go to https://plone.org/download/releases/5.2.3

Plone 5.1.7

Plone 5.1.7 is a bugfix release of 5.1. Note: this is the last release in the 5.1 series.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.1.7/versions.cfg.

Some highlights are:

  • Integrate Plone20200121 hotfix.
  • Security: depend on Products.isurlinportal. Version 1.1.0 has hardening against white space.
  • plone.recipe.zeoserver: Windows fixes
  • mockup and plonetheme.barceloneta: various frontend fixes, including translations
  • Lots of translation updates
  • Lots of bug fixes in many packages.
  • plone.namedfile: Range support
  • plone.scale:
    The mode argument replaces the old, now deprecated, direction argument.
    The new names are contain or scale-crop-to-fit instead of down,
    cover or scale-crop-to-fill instead of up and scale instead of thumbnail.
  • plone.supermodel: added support for choices of integers for improved registry.xml export.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group.

For detailed changelog, go to https://plone.org/download/releases/5.1.7

Plone 4.3.20

Plone 4.3.20 is a bugfix release of Plone 4.3. Note: this is the last release in the 4.3 series.

Note that support for Python 2.6 was dropped a while ago. It might still work, but you should use Python 2.7.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/4.3.20/versions.cfg.

Some highlights are:

  • Integrated PloneHotfix20200121 for increased security.
  • Moved the security check if a url is in the portal to a small separate package: Products.isurlinportal. You can immediately use this on Plone 4.3 and higher. Keep an eye on updates for this package: newer versions will increase the security. Often the impact of fixes is too small to warrant a real security hotfix package, but we want to do more regular fixes here.
  • Use Products.isurlinportal 1.1.0 with security hardening against whitespace: https://github.com/plone/Products.isurlinportal/issues/1
  • Removed broken X-XSS-Protection header from classic theme and unstyled theme.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group. Notify these events when principals are added or removed to a group in ZODBGroupManager. See https://github.com/zopefoundation/Products.PluggableAuthService/issues/17
  • z3c.autoinclude: When environment variable Z3C_AUTOINCLUDE_DEBUG is set, log which packages are being automatically included.

For detailed changelog, go to https://plone.org/download/releases/4.3.20

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

Data-driven documents with Volto and Plone

Posted by PLONE.ORG on November 09, 2020 12:05 PM

Executive Summary

Innovations in the Plone environment continue to expand the platform's capacities to provide scalable solutions. The BISE project realized by Eau de Web demonstrates the potential of joining Plone's world-class CMS back-end with the flexibility of the Volto front-end user interface. In this case, the integration enabled a highly flexible display of data-driven dashboard information with factsheets that are editable through the web by content managers. Get the full story below.

An Innovative Approach to a Known Problem

The BISE website (biodiversity.europa.eu) is one of the thematic environmental information websites run by the European Commission (DG-ENV) in cooperation with the European Environmental Agency. It used to run on Plone 4, but as part of the obligatory upgrade to Plone 5, we (Eau de Web) have migrated to a system based on Plone 5 and Volto 8.

Product of almost one year of work, as it stands right now, BISE is yet another example of the benefits of bringing closer the frontend development world to the solid foundation provided by Plone. The highlights of the BISE portal are, probably, the Country Biodiversity Factsheets, available in the Countries section.

Screenshot_20201029_155123.png

A mix of data, charts, and specially created visualizations would surely imply, in classic Plone development fashion, a dedicated content type and Python code, but in our case, the content type has been created through the web and holds little logic or knowledge about the end result. Instead, the factsheets are fully editable and composable through the web, based on somewhat generic Volto blocks: “plain” rich text blocks, chart blocks, a couple of “data table” types, and a “columns block” to enable composing these in a more complex but mobile-responsive layout. For even more flexibility, we’ve created a generic block styling framework that can be used for any Volto block and enables basic styling, such as alignment, text colors or to choose from a palette of predefined styles.

Everything, starting with the data definition (which can be provided either as plain CSV documents uploaded as Files in Plone or as SQL queries for a custom content type that communicates with the SQL server via REST) is completely generic and editable through the web, as part of the Volto UI.

The newly arrived “Volto dexterity-based content layout” bridges the gap between the goodness of Volto blocks and the templating power of content types. Add on top of that a thin layer that automatically adjusts templated content data (text, charts, data tables) to the context metadata (in our case, country code) and we get a high productivity environment that empowers the content experts to provide significant enhancements to the website without relying on the developers. 

plotly.png

Volto-Slate and Other Contributions

One of the key components of this website is also invisible due to its ubiquity: a new rich text editor, the volto-slate. The Biodiversity website relies heavily on the text editor: the website has a lot of textual information and its editors need to rely on a feature-rich editor. Also, the country factsheets use “templated text”, predefined (but editable) text that embeds numeric facts that come from the data connectivity framework. These “data-connected text entities” need to be fully configurable and styled by the rich text editor.

Early in the development cycle and having the experience of a prototype built with Volto’s default Draft.js editor, the development team decided to abandon it and reimplement a drop-in replacement with another library: SlateJS. Rather than providing an editor with a default configuration, SlateJS is different: it comes with the building blocks and a rich React api to enable building any kind of richtext editor. The Volto blocks and its full-page editing philosophy are rather specific, so we needed a library that can be used for this kind of deeper integration.

In the end, the volto-slate editor reached a stage where it can provide a level of integration that exceeds that of Volto’s built-in editor: improved block splitting and joining, better pasting with block splitting, automatic image and table block extraction and better overall handling of lists and sublists, just to name a few. It is now a stable target and provides plugin primitives for other rich-text integration tasks, sprouting other Volto addons dedicated to it: volto-slate-zotero, volto-slate-metadata-mentions, etc.

Screenshot_20201028_150648.png

More additions to the overall Volto ecosystem

In fact, volto-slate constitutes a substantial part of the overall development effort, but it wasn’t the only contribution that the BISE project made to the overall Volto ecosystem. Among them we can list:

  • Major improvements to the Volto addons architecture
  • Various bug fixes and enhancements to Volto core
  • Contributions to various Volto-connected projects, such as plone.restapi, the yeoman-based Volto app template generator and the styleguidist documentation
  • Many addons available as open source from the EEA github repo:

dotted-table-chart.png

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone. Through the freedom that they provide, Volto blocks are a foundation for innovation that enables Plone to step in line with the latest state of the art for web development.

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone.

Eau de Web is a software company specialising in Web development. The drivers of our activities are the open-source culture and the use of open standards due to the benefits of code reuse, openness in participating in projects worldwide, peer review and contributions from a wide community of developers. Our group has used Plone for web development from the early days, Eau de Web is a Plone foundation member, and our team members continue to demonstrate a long-term commitment to share and contribute the knowledge, software and resources to benefit others through Plone’s open source community.

Mikel Larreategi becomes a member of the Plone Foundation

Posted by PLONE.ORG on October 29, 2020 03:02 PM

The Plone Foundation welcomes a new member, after unanimous confirmation by the Foundation's Board of Directors on October 22, 2020.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.


Mikel Larreategi

erral.jpgMikel started working with Plone with the 2.1 version after joining CodeSyntax. Mikel attended several Plone Conferences (Naples, Washington DC, Budapest, Bristol (twice), San Francisco, Arnhem, Bucharest, Barcelona and Ferrara, and, between other community activities, maintains the translations of Plone into Spanish and Basque.

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Plone Conference 2020 - A Sneak Peek

Posted by PLONE.ORG on October 28, 2020 08:04 PM

Over 65 Talks Submitted

The topics range from Plone, Zope, Volto and Guillotina to Python and Pyramid, and from fancy JavaScript to cool case studies. There are 30 and 45 minute talk slots, as well as lightning talks. 

The talks are targeted at a variety of different audiences such as designers, developers, integrators, users, and beginners alike. There will be something for everyone and the schedule will be planned so, that it is easy to find suitable talks for every need!

Volto Talks Galore

Volto, Plone's React front end, continues to gain interest, and there will be plenty of talks to catch you up on the latest developments:

  • Volto FTW
  • Building Volto Addons
  • Volto: A Journey Towards Personalisation
  • Plone and Volto in a Jamstack Project

What's in Store for Plone 6

Many people are helping move Plone 6 closer to reality - here are some of the topics they will be presenting:

  • Plone 6 Theming: Barceloneta
  • Plone 6 Theming with Diazo
  • Modernizing Plone's Classic UI

Agile, Patterns, Projects

We will dive into what goes into a successful project - from project management to technical architecture to building community:

  • Second Guessing the Single-Page-App Pattern
  • Asking Questions for the Benefit of your Future Self - Growing with the Plone Community
  • Agile Race To Zero
  • The Effectiveness of Open Source to Achieve a Common Heritage for Cities

And More!

Here are a few more intriguing-sounding talks:

  • Guillotina: Real Use Cases & Roadmap
  • State of Pillow
  • Collaborating With Orchid Data
  • Rapidly Building An Extensible Corruption Tool
  • Oh, the Places We've Been! Plone 2001 - 2020

There will be something for everyone!

Get you Plone Conference tickets now at https://2020.ploneconf.org/

New print-css.rocks release

Posted by Andreas Jung/ZOPYX on October 16, 2020 07:45 AM
New print-css.rocks release

Earlier this month, I released an update of my PrintCSS Portal print-css.rocks.

Thiis update includes support for new tools like

  • WeasyPrint
  • Typeset.sh
  • PagedJS
New print-css.rocks release

I also want to mention the portal printcss.live by Andreas Zettl. printcss.live is a very nice and supplementary addition to the PrintCSS community because it allows you to test tools directly within the browser. Andreas Zettl also maintains a list a PrintCSS articles on medium.com.

Blog moved to Ghost

Posted by Andreas Jung/ZOPYX on October 14, 2020 04:59 PM
Blog moved to GhostGhost Logo
Blog moved to Ghost

After almost 12 years I moved my personal blog blog.zopyx.com from Plone 4 to the Ghost blogging platform. You may ask why? First, my old Plone 4 installation reached end of lifetime a long time ago and it was time for moving the content to a different system. I investigated a lot of time into options like static site generators like Eleventy, Huge and others but I was not very much happy with them. I just wanted a simple system where I can add and publish content easily.

The migration from Plone 4 to Ghost was a bit painful. First, I tried to use plone.restapi or collective.jsonify but the Plone 4 site gave me a bunch of headaches and errors. So I decided to create a static copy of the original bog using httrack. With some Python code I was able to extract the content part from the HTML markup of each document and some necessary metadata like publication date, title, description etc. All extracted data was written to a JSON file.

Ghost itself provides some JSON format as import/export format. Due to limited resources I hired a Ghost developer (for a very small budget) that converted my JSON format into the Ghost JSON format.

Then setting up Ghost (running in a Docker container) behind Apache was more or less straight-forward. As with every new system: you need to learn and try things out.

For the moment, I am happy with the result and outcome. There are clearly some issues like a better theme or some CSS fixes to be implemented.

I hope to blog again more in the future.

Plone Conference 2020 - Submit a Talk and Get Your Tickets!

Posted by PLONE.ORG on October 14, 2020 04:17 PM

Welcome to Plone Conference 2020 from where ever you are!

This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org. The conference will be held from 5th to 13th December 2020. No matter where you are, you can participate.

Some preliminary information (might change still):

  • Training will happen over the weekend - December 5 - 6.
  • 4 days of talks + 1 of open spaces - December 7 - 11.
  • Sprint - December 12 - 13.

Call for papers is open, please submit your talk proposal

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences too

https://2020.ploneconf.org

Important dates

Stay tuned for more information! Follow @ploneconf and #ploneconf2020 on Twitter, Instagram, and TikTok!

Zope 5.0 released

Posted by PLONE.ORG on October 10, 2020 06:55 PM

Zope is the original open source web application server written in Python, featuring a component architecture and an object-oriented, hierarchical data model.

This release

  • drops support for Python 2.7 and 3.5
  • drops support for Zserver
  • adds support for the newly released Python 3.9

Although this release is a major one, it should be possible to use it without any changes if the following preconditions are met:

  • Your project runs on the latest Zope 4 release on Python 3.6+ and does not use ZServer.

  • Neither your tests nor running your Zope instance shows DeprecationWarnings.

Additionally, this release contains some minor features and bug fixes that have been backported to Zope 4 where possible.

As Zope 5 is now released, Zope 4 is put into bugfix mode until 2021-12-31.
(See the roadmap for details: https://zope.readthedocs.io/en/latest/roadmap.html 3)

For details about this release see the what’s-new-section and the changelog:

To install the new version see https://zope.readthedocs.io/en/latest/INSTALL.html

Plone 5.2.2 released

Posted by PLONE.ORG on September 27, 2020 01:00 PM

Plone 5.2.2 is a bug fix release of Plone 5.2. Release Manager for this version is Maurits van Rees.

Download Plone 5.2.2

Note: We want to do releases more often, to avoid having very big changes between two bugfix releases. Plone 5.2.3 is expected in October 2020. We announce pending releases for testing on https://community.plone.org, and we announce a final release there once it is out of the pending phase, even when the installers are not ready yet.

Installers are being made, so not all links below will work yet. Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.2/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Release notes for Plone 5.2.2

This release packs a lot of changes: the previous release was from January 2020. Some highlights are:

  • Zope was upgraded from 4.1 to 4.5. This means the support for WebDav is back. And the logic around template engines got updated. There should now be fewer differences in behavior between various kinds of templates (Zope/Acquisition-aware or not) or which template engine is used (zope.pagetemplate or chameleon). There were some problems due to this change, but all should have been fixed now.
  • ZODB goes from 5.5.1 to 5.6.0. These  updated packages are needed for zodbupdate and ZEO. So if you are doing low-level transactional stuff, please take a look at its changes.
  • plone.recipe.zope2instance: changes for relstorage, sqlite, webdav, Windows.
  • Fixed problems running Plone on Windows. You may want to keep an eye out for newer releases of plone.recipe.zope2instance. See especially this issue: https://github.com/plone/plone.recipe.zope2instance/issues/144
  • Mockup and plone.staticresources: lots and lots of improvements and bug fixes in styling and javascript.
  • Big push in fixing various internationalization issues.
  • Integrated PloneHotfix20200121 for increased security.
  • Moved the security check if a url is in the portal to a small separate package: Products.isurlinportal. You can immediately use this on Plone 4.3 and higher. Keep an eye on updates for this package: newer versions will increase the security. Often the impact of fixes is too small to warrant a real security hotfix package, but we want to do more regular fixes here.
  • Use Products.isurlinportal 1.1.0 with security hardening against whitespace: https://github.com/plone/Products.isurlinportal/issues/1
  • plone.app.discussion: Extended existing review workflow by states rejected and spam.
  • plone.app.theming: Add custom CSS settings to theming control panel. You now have a textarea where you can simply add CSS without needing to override things or play with less/scss.
  • plone.namedfile and several other packages: support for Range requests.
  • plone.outputfilters has a possibly breaking change: change the image caption template to use <figure> and <figcaption>. If you want to change this, there is a new ImageCaptioningEnabler utility.
  • plone.restapi was upgraded from 6.1.0 to 6.13.7, with lots of new features.
  • plone.scale: the method signature for scaling was made clearer. The mode argument replaces the old direction argument. Direction is still supported, but now deprecated. The new name for direction down is: mode contain or scale-crop-to-fit. The new name for direction up is: mode cover or scale-crop-to-fill. The new name for direction thumbnail is: mode scale.
  • plone.testing 8.0.1 drops the z2 extra. If you need this, use the zope extra instead.
  • Products.CMFPlone: Add markdown extension settings to markup control panel.
  • z3c.autoinclude: When environment variable Z3C_AUTOINCLUDE_DEBUG is set, log which packages are being automatically included.
  • zope.interface was upgraded from 4.6.0 to 5.0.2. This has performance fixes that may have backwards incompatible changes. This makes the inheritance and method resolution order for interfaces more logical and more in line with how it works for 'normal' classes. In corner cases there can be subtle differences when there are several utility or adapter registrations for similar interfaces. For example, when looking for browser view X and there are two such views, Zope looks for the most specific matching interface, and this may have changed.

Changes

For detailed changelog, go to https://plone.org/download/releases/5.2.2

The First Plone Steering Circle Meeting

Posted by PLONE.ORG on September 26, 2020 05:00 PM

The Initiative

Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.

  1. A series of Steering Circle meetings will be held to discuss the issues and plot a way forward.
  2. Beginning at the November Plone Conference, the Board will set up a series of open discussions about the organizational structure of all our projects. All community members will be welcome!

This initiative is described in more detail in the Board‘s statement on the Plone governance process.

The First Meeting

The first Steering Circle meeting was held on Zoom on 22nd September 2020. Representatives from each community team were invited and 22 people from around the world attended. Discussion items included:

  • The purpose, process, and schedule of the Steering Circle
  • Status of the teams - are any inactive, should new ones be formed
  • The Zope team's concerns about expectations regarding meetings, minutes - no, this is not necessary
  • The Framework Team vs. the Roadmap Team vs. a proposed new Plone Core Strategic Team
  • How to foster transparent communication throughout the community - Plone, Zope, Guillotina, Volto

The meeting minutes are posted on community.plone.org.

Volto 8 Released With Exciting New Features

Posted by PLONE.ORG on September 25, 2020 05:38 AM

Volto is Plone's snappy, modern React front end, and the Volto team has added two major new features to the latest release.

Through the Web Custom Types

Beginning with Plone 4, site admins have been able to create customized content types easily in the browser using Plone's standard administrative UI. Now, this feature has been added to Volto. You can:

  • Add a custom content type
  • Edit its fields and fieldsets, help text, etc.
  • Select from numerous predefined field types (selection, text field, date, file upload etc.)
  • Rearrange fields on the edit form
  • Abstract content type elements into behaviors which can be used on other types

In short, it's possible to customize a content type in every possible way, and the resulting type is automatically searchable and addable.

Through the Web Custom Type Layouts

Blocks are Volto's way of adding and organizing various elements on a page. They can be text elements, images, videos, dynamic lists, video embeds, or whatever you decide to develop. In Volto 8, you can combine the features of blocks editing with your newly created content type, vastly enhancing the functionality and customizability of your custom content - all in the browser with just a couple of clicks. 

Try Volto Today

Download Volto from Github or kick the tires on the Volto demo site. 

Matthew Wilkes Publishes Book: Advanced Python Development

Posted by PLONE.ORG on September 15, 2020 04:43 PM

As some Plone community members know, one of our own, Matthew Wilkes, has been hard at work writing a book for the last year. We're very excited to announce that it has been published! It is Advanced Python Development: Using Powerful Language Features in Real-World Applications and it can be purchased at Matthew's website, Apress or Amazon.

The book follows the design and build of a real-world application example from prototype to production quality. In this context, it explains Python language features that aren’t routinely covered and introduces techniques that demonstrate large scale system design and development processes. Plus there are useful asides, best practices and library recommendations galore.

Matthew specializes in Python development, especially in the areas of web security and performance. He has been developing Python web applications for 15 years and has been very active in the open-source community in general and the Plone community in particular.

Matthew began using Plone in 2004 when it was at version 2.0. He has worked on various aspects of the core Plone software and was on the Plone framework and security teams through the Plone 4 era. He led the security team for over four years, modernizing security team practices by instituting unit testing of fixes across all versions, cooperation with Red Hat and other downstream vendors, and a pre-announcement policy. He has also contributed to the security of Zope, Django CMS and Pyramid, including recently rewriting Pyramid’s CSRF protection. This all gives him a wider view of security as a process rather than just deep knowledge of one system.

Matthew served as the administrator for Plone's participation in Google Summer of Code for many years. He has also served on the Plone Foundation's Membership Committee and Board of Directors, and he was instrumental in helping to overcome the many legal and logistical hurdles to bringing Zope under the Plone Foundation's umbrella.

Matthew is a welcome and frequent presence at Plone conferences and sprints and has been known to be involved in adventures in late night drinking. For example after the party at the first Bristol conference, Matthew was instrumental in arranging for the pub where many Plonistas were hanging out to continue serving them after official closing time. The party continued until 5 in the morning and we learned later that the place took in £16,000 that night. Matthew also collaborated with David Glick on the classic zodb.ws April Fool’s Day launch in 2011 when they managed to get a hacked-up version of the ZODB running in the browser - a prank which required both technical chops and a particular sense of humor.

Congratulations to Matthew!

Plone Conference 2020, November 14 - 22 Will Be an Online Event!

Posted by PLONE.ORG on September 11, 2020 08:23 PM

Welcome to Plone Conference 2020 from where ever you are!

This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org.

Some preliminary information (might change still):

  • Training will happen over the weekend - November 14 & 15.
  • 4 days of talks + 1 of open spaces - November 16 - 20.
  • Sprint - November 21 & 22

The conference will be held from 14th to 22nd November 2020. No matter where you are, you can participate.

Stay tuned for more information soon!

Call for papers is open, please submit your talk proposal at
https://2020.ploneconf.org

And one more thing...

Plone Conference 2021 will be held in Namur, Belgium!

Stay tuned for more information!

Plone Conference 2021 teaser

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\[email protected]]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'[email protected]': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'[email protected]': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone [email protected]'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Plone Foundation Launches Governance Initiative

Posted by PLONE.ORG on August 20, 2020 06:07 PM

Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.

  1. Every two months beginning in September, a series of Steering Circle meetings will be held to discuss the issues and plot a way forward. Representatives from all Plone teams are being invited to the Steering Circle meetings. People who are not affiliated with a team may request an invitation from the Board.
  2. Beginning at the November Plone Conference, the Board will set up a series of open discussions about the organizational structure of all our projects (Plone, Zope, Guillotina, Volto, REST API, etc.) All community members will be welcome to attend and virtual participation will be possible.

This initiative is described in more detail in the Board‘s statement on the Plone governance process. Thanks to the Board for proposing this and we all look forward to a series of fruitful discussions!

Great Progress on Volto at the 2020 Beethoven Sprint

Posted by PLONE.ORG on July 24, 2020 01:40 PM

Everybody in the Plone community loves sprints. Not only because of the software we enhance or the code we write. The most crucial ingredient of a great sprint is meeting the people you like and that you haven’t seen in a while. This is harder to do in these coronavirus times, but the Beethoven sprint, held remotely from May 27th through the 30th, succeeded.

There were 34 participants, including substantial representation from kitconcept, RedTurtle, and Eau de Web. They used Discord voice channels throughout the day for a real sprint experience. The channels functioned like tables in a sprint room, where sprinters can go from one table to another to hear what people are discussing. The organizers also used Zoom calls for daily standups and Remo.co video calls in the evenings, which are structured like a bar. Sprinters brought their favorite beverage, good food, relaxed on a couch, and had good conversations in the evenings!

In addition to socializing, lots of work got done. The sprint was all about Volto, which is the new React front end for Plone 6. Highlights included:

  • Bug fixes, date widget improvements, a recurrence widget, a sort function, and work on widget reusability
  • Discussions and progress on a new Volto add-ons architecture based on slots
  • Improvements to Volto's multilingual story, including a test fixture and a translation for Romanian
  • Improvements to the Cypress accessibility test coverage
  • Discussions about Volto's theming story, including how to separate the CMS and public-facing themes
  • A form builder prototype, as well as lots of form builder discussions
  • A Volto listing block that includes sorting, view templates, and a table view
  • Progress on a Dexterity content types control panel for Volto, including a schema editor and RestAPI endpoint
  • Additional Volto training materials, as well as training test drives
  • Volto marketing and onboarding work, including a Volto website and screencasts
  • Discussions and improvements to the Plone backend, including a plone.restapi release
    and work on the thorny problem of a Dexterity site root object
  • Plone 6 roadmap discussions

Thank you to everyone who attended - Plone 6 is getting closer to reality!

Product Lifecycle Management with Plone

Posted by PLONE.ORG on July 14, 2020 07:02 PM

Plone at PNZ Produkte GmbH

PNZ Produkte GmbH is a niche manufactory of oil-based paints and coatings made from renewable resources. Next to its company brand product line, a substantial amount of business is generated from private label (PL) sales. These PL products are offered as a turn-key service: including promotional material, product information sheets and packaging. For some private label customers, PNZ also takes care of fulfillment to end consumers.

A typical finished product is derived from:

  • A Base Recipe which determines the ingredients, production methods, usage safety indications and applicable mandated regulations (e.g. GHS)  
  • A Master Product which defines the application specifications, usage instructions, packaging sizes, pigments and other general use attributes
  • A Finished Product which includes branding characteristics, marketing descriptions, article EANs and other customizations

Since 2015, Plone has been used as the engine for creating and maintaining product specifications. About 20 employees in multiple departments interact regularly with the software. The project received a “Digitalbonus” innovation award from Germany's Landesamt Oberbayern.

PNZ Product Settings Page

A Typical Product Lifecycle

When a new product is formulated, has passed its application and safety testing phase, and is ready to go in production, its specifications will be split in three separate information layers. Custom content types in Plone connect these layers and manage their contents. Plone's ability to quickly prototype content types through-the-web makes it easy to design an initial structure and refine it based on user experiences. Each information layer consists of 30 to 50 individual fields and can be extended as needs change over time. Connecting different types of data from dynamic and external sources is relatively simple as Plone offers a large variety of edit/display widgets out of the box. 

At each hierarchical level, new content undergoes a quality and regulations compliance control check before going through the Plone publishing workflow. This makes the document available for use in lower layers. Any base recipe can potentially power several master products, which in turn results in a large number of finished products. The current system powers some 25,000 individual articles or SKU's. 

At the lowest level, a "Finished Product" content type includes product-specific overrides and any information not inherited from the recipe and master product layers.

PNZ Product Information Flow

Since products may be offered to multiple geographical markets, all information must be translated. Plone's built-in multilingual capabilities make it possible to have each individual document available in several languages. Fifteen languages are currently in use within the PNZ system. Each field in a content type can be designated as translatable or multilingual, so that updates of multilingual fields can automatically propagate throughout the various translations.

Automating Content Creation

The information contained in its product lifecycle management system allows PNZ to automate the generation of a variety of ready to use content. Plone provides the BrowserView mechanism, used in conjunction with Zope Page Templates (ZPT), to determine what should be included in the content and how it should be rendered. PNZ Wrappers with inDesign and XML from Plone

PNZ creates colortone samplers and product information sheets, with a customized layout for each brand, using the tools provided by Plone. The documents then undergo a final conversion to PDF using “Produce & Publish” software, created by longtime Zope/Plone community contributor Andreas Jung at Zopyx.

In the packaging process, Plone powers the creative design of all product labels. A BrowserView is used to draw data from multiple content types and the company’s ERP system. This will generate a set of XML files for each product and its SKU variants, plus related dynamic images such as EAN codes, colortones, certifications and GHS warning signs.

PNZ Public Site With PloneThe XML files and images are then read into Adobe InDesign templates by the company's creative designers, resulting in a professionally designed wrapper for each SKU that always contains accurate information and complies with industry standards and specifications.

Other types of content include the generation of raw HTML and images for use in a PlentyMarkets shop, and the creation of Product Information Management (PIM) CSV export files for large customers such as DIY store chains. Finally, using plone.restapi, the system powers the product catalog of PNZ's soon to be relaunched public facing website which of course also uses Plone!

A Very Successful 2020 Python Web Conference

Posted by PLONE.ORG on June 26, 2020 03:09 PM

June 17-19, Six Feet Up hosted the 2nd Annual Python Web Conference using their newly developed all-in-one virtual event platform, LoudSwarm. Focusing on Python for the Web, the conference featured talks and trainings from several members of the Plone community.

The conference had a total of 266 attendees participating from 37 countries and 14 timezones. A unique aspect of the registration was that for each ticket purchased, a ticket was donated to a developer residing in a developing country. Those that received a free ticket would not have been able to otherwise attend, and they were able to gain valuable knowledge on the state of Plone and Python on the web.

Being a completely virtual event, several accommodations were set up to make the event as interactive as possible. Attendees could join the Gallery after each talk to ask the presenter questions and have discussions about the talk topic. Slack was also heavily used to foster discussions and interaction among the attendees. Evening Socials were created to network, play games, and keep the fun going after the day’s talks were done.

Having a Plone presence was great for the attendees who had never heard of Plone, and provided many with the opportunity to see Plone again in a new way. Many great discussions were had around new Plone features, customization tricks, and fond memories of past Plone Conferences.

Recordings of the talks are currently only available to registered attendees. Post-video access tickets are still available if you weren’t able to attend the live event. All talk and tutorial videos will be made publicly available on YouTube in September.

Plans for Python Web Conf 2021 are already underway. Visit https://2020.pythonwebconf.com to learn more about this year’s event.

Zope and the Plone Foundation

Posted by PLONE.ORG on June 25, 2020 02:59 PM

Plone was very early to open source governance - the Plone Foundation was one of the first 501(c)(3)s to be established to protect open-source software. In the years that have followed it's become more common for software to be protected under an umbrella organization, but that wasn't possible in the early days.

Zope started off as the property of Digital Creations, which became Zope Corporation. But in the early 2000's there was a desire to move the protection of Zope to a community effort following the same model, so a new non-profit was formed.

This worked well for ten years, but by the mid 2010's there were significantly fewer active developers of Zope and most were not interested in the administrative tasks required to maintain a non-profit corporation. Over the years, the Zope Foundation fell behind in its regulatory compliance until it became inactive. It still owned the copyright to Zope, as well as some funds, but it had very limited rights over them.

The Plone Foundation's mission to protect Plone implies a responsibility to protect Zope, as Plone depends on Zope for much of its core infrastructure. Informal discussions began around this time about Plone taking over the role of the Zope Foundation, but no substantial progress was made.

As the years passed, the need to solve the regulatory problem became more pronounced. It was clear that the Zope Foundation would not restart operations in any meaningful sense, but Zope development was undergoing a resurgence thanks to the need to port to Python 3.

At the 2018 Plone conference in Tokyo, after quite a few whiskies, people resumed talking about this problem, and we decided to try to solve it again. Because one of the biggest blockers was a lack of enthusiasm in the Zope community with regards to dealing with the bureaucracy, some people who were members of both Foundations reached out to the IRS, the Delaware Department of State and some experts in US Corporation law to plan a way forward.

A few months later in early 2019, the Zope Foundation held a special general meeting to pass a motion on the future of the Foundation. The membership mandated the board of directors to transfer all assets to the Plone Foundation as a donation, and to formally close the Zope Foundation. Later that same day the board of directors voted to donate all property of the Zope Foundation, including copyright in the Zope software, to Plone.

Since then, Plone community members have been working on the bureaucratic aspects of this move. We've harmonized contributor agreements to improve international enforceability, and work is underway to ensure that Zope contributors are properly represented in Plone's democratic structure before the next internal election in November. Zope contributors can already apply to be members of the Plone Foundation, and many are, but we're looking to streamline the process for former Zope Foundation members.

The Zope and Plone Security Teams have already been integrated, and now we need Zope representatives on the Membership Team. Any Zope people who are interested in helping out should please contact Érico Andrei.

© Plone Community and friends.

Blog entries aggregated on this page are owned by, and represent the opinion of the author.