Plone Logo

Planet Plone - Where Developers And Integrators Write

Plone 6 Beta Released

Posted by PLONE.ORG on August 04, 2022 06:15 PM

Good news: the first beta release of Plone 6 has arrived! The release manager for this version is Maurits van Rees.

Thank you to everyone involved!

Read more about the upcoming Plone 6.

Highlights

Major changes since 6.0.0a6:

  • Products.CMFPlone:
    • Removed our expressions patch. This was a patch to avoid some too strict checks by Zope / Products.PageTemplates. But in Plone 6 it should be fine to be stricter. The STRICT_TRAVERSE_CHECK environment variable is no longer read.
    • Initially open accordions in resource registry. Hide via JS when no errors occur. This makes it possible to fix a breaking error.
    • Resource bundle dependency on multiple comma-separated names.
  • plone.staticresources:
    • Register jquery and bootstrap globally. This helps a lot for add-ons.
    • Fix TinyMCE link/image modals.
  • plone.dexterity: Remove long deprecated imports and fallbacks.
  • plonetheme.barceloneta: Update to Bootstrap 5.2.0 (released this week).
  • plone.scale: Fix to ensure that when a scale that was registered using pre_scale is later actually generated by get_or_generate, it is stored with the same uid as the placeholder info that was stored by pre_scale. This avoids an issue where the same scale was generated repeatedly.

Volto frontend

The default frontend for Plone 6 is Volto. The latest release is 16.0.0-alpha.15.

This contains the much-anticipated integration of the volto-state editor. See the changelog.

Slate editor - toolbar example

Slate toolbar

Slate editor - editing links

Links editing

Python compatibility

This release supports Python 3.7, 3.8, 3.9, and 3.10.

Support for Python 3.10 is still a bit provisional, as our testing infrastructure needs some updates before we can fully test this.

But your friendly Plone Release Manager is using Python 3.10 for local development, so it should be okay.

Installation

For installation instructions, see the documentation.

This documentation is under development, but this should get you up and running. No worries.

For technical wizards who want to jump straight in, here are two important links:

Plone 6

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Try Plone 6 Beta!

For installation instructions, see the documentation.

See Plone 6 in action at https://6.demo.plone.org/

Read more at the community forum:
https://community.plone.org/t/plone-6-beta-1-released/15485

Volto recipe for footer actions managed as site content

Posted by PloneExpanse on July 18, 2022 02:10 PM
Managing the Footer as content is one of the common tasks on a Plone / Volto website. One typical approach is to designate some root folder, let’s say footer-links as a container for Link instances, and use those links as shortcuts to dedicated pages. So, a footer component may look like this: import React from 'react'; import { getContent } from '@plone/volto/actions'; import { useSelector } from 'react-redux'; import {UniversalLink} from '@plone/volto/components'; const Footer = () => { const footerLinks = useSelector((state) => state.

Become a Sponsor to Plone Foundation

Posted by PLONE.ORG on July 09, 2022 02:45 PM

Plone and the related family of technologies (Volto, Zope, Guillotina) are very strongly community-oriented.

Development often takes rapid leaps in sprints, which have been a defining characteristic of our history. Sponsorship allows more of these sprints to take place.

There are many options, starting from one-time payments to monthly subscriptions, and starting from as low as 5$.

In addition to supporting a good cause and helping Plone Foundation, you can have a shoutout at Twitter, a GitHub badge or your name listed at Plone.org.

Sponsoring Plone is very easy and you can do it at https://github.com/sponsors/plone

Thank you!

Plone 6.0.0a6 Released

Posted by PLONE.ORG on July 08, 2022 02:00 PM

Download and installation: https://plone.org/download/releases/6.0.0a6

The sixth alpha release of Plone 6 contains various fixes and improvements. Read more about the upcoming Plone 6.

Thank you to everyone involved!

Highlights of Plone 6 alpha 6

Major changes since 6.0.0a4:

  • Wait, is there no alpha 5? No. We created an alpha 5 of Products.CMFPlone, which is the main package. Before wrapping up a full Plone release, an important problem was fixed, which required a new release of this package. So now we jump from alpha 4 to alpha 6.
  • The versions.cfg file on dist.plone.org now only contains versions needed for core Plone. We used to pin some often used "ecosystem" packages like plone.app.tiles. Those pins are now in versions-ecosystem.cfg. Several other pins, mostly for tools like zest.releaser, are now in versions-extra.cfg.
  • plone.app.caching: Remove unmaintained Split-View profile.
  • plone.app.content:
    • Remove unused container and item modules.
    • Remove unused IReindexOnModify.
  • plone.app.i18n, plone.i18n, plone.app.multilingual: Use SVG Flags in Language Selector, update name of flags, and prepare to use the Icon Resolver.
  • plone.app.upgrade:
    • Removed old code, aliases and dependencies. We only support upgrading from Plone 5.2 Python 3.
    • Upgrade profiles of core Plone modules to specific versions in the upgrade to alpha 5. This way, when later upgrading a site from 5.2 to 6.1, these initial upgrades are done at the end of the alpha phase, instead of completely at the end. This should avoid some surprises.
    • Add image_scales catalog metadata column. Update all brains to get this info. Since this takes long on large sites, you can disable this with an environment variable: export UPDATE_CATALOG_FOR_IMAGE_SCALES=0. In that case, you are advised to add the image_scales column manually to the catalog later.
  • Products.CMFPlone:
    • Remove Archetypes specific code: isIDAutoGenerated, PloneFolder, DublinCore.py.
    • Moved discussion Key to plone.app.discussion.
    • Added customisable batch_size for redirects controlpanel.
    • Add option to use TinyMCE in inline-mode.
    • Add support for images in default search page.
    • Enable auto include of styles to the TinyMCE formats menu. The file has to be named tinymce-formats.css and known by TinyMCE.
    • Add image_scales to catalog metadata.
    • Sort addons by title
    • Show more information of broken relations
    • Show link to the Volto-migration (@@migrate_to_volto) in the view @@plone-upgrade when the option is available.
    • SVG image as default Plone logo.
  • plone.api: In content.get_view and portal.show_message do not require the request parameter to be specified. If not specified, fallback to the global request.
  • plone.app.contentmenu: Reimplement dropout toolbar submenus and collapsed icons.
  • plone.app.querystring: Add negation-query operators plone.app.querystring.operation.string.isNot and plone.app.querystring.operation.selection.none.
  • plone.app.robotframework: Add keywords for making sure an element is visible before clicking: Wait For Element, Wait For Then Click Element, Wait For Then Click Invisible Element.
  • plone.autoinclude: Raise an exception when a module is not found. When environment variable AUTOINCLUDE_ALLOW_MODULE_NOT_FOUND_ERROR=1 is set, we log an error and continue. To accept ModuleNotFoundError only in specific packages, use a comma-separated list of project names, with or without spaces. See issue 19.
  • plone.base:
    • Add image srcset's configuration including JSON schema definition to imaging-controlpanel.
    • Enable images in search results by default.
    • Add inline mode to tinymce config.
    • Move Products.CMFPlone.utils._createObjectByType to here as utils.unrestricted_construct_instance.
    • Add images interface with IImageScalesAdapter and IImageScalesFieldAdapter.
    • ulocalized_time: accept a string argument to long_format.
      For example: ${a} ${d} hello guys ${b} ${Y}.
      Taken over from experimental.ulocalized_time.
  • plone.scale:
    • Pre scale: store non-random uid to prepare space for a scale. You call pre_scale to pre-register the scale with a unique id without actually doing any scaling with Pillow. When you later call the scale method, the scale is generated. You can still call scale directly without first calling pre_scale.
    • Mark AnnotationStorage with safeWrite fromplone.protect.
  • plone.namedfile:
    • Creating a tag no longer generates the actual scale. The scale is only created when a browser really requests it.
    • Add @@images-test page for Editors. This shows various variants from the image field of the current context. It shows a list of stored scales. It allows purging the stored scales.
    • Add picture method to ImageScaling.
    • Removed marking request for disable CSRF protection
    • Add additional infos in scale storage only if missing.
    • Register adapter for image fields to the new image_scales metadata. Use this in the image_scale view to get images from a list of brains.
  • plone.outputfilters: Add image_srcset output filter, to convert IMG tags into PICTURE tags with multiple source definitions as defined in the imaging control panel.
  • plone.staticresources:
    • Integrate bootstrap and jquery bundles with module federation. They now live in the bundle-plone directory.
    • Update toolbar toggler.
    • Update to latest Mockup with module federation.
    • Add image full screen support through full screen API.
    • structure pattern fixes.
    • Refactor pat-recurrence.
  • plonetheme.barceloneta:
    • Example for extra styles that get automatically populated to the TinyMCE formats menu added.
      *Reimplement dropout toolbar submenus and collapsed icons.
    • Create CSS variables for Plone colors.
    • Update to Bootstrap 5.2.0-beta1 and make use of CSS variables.
    • Global sections: add support for css variables and color modes barceloneta, dark, light
  • plone.volto:
    • Add form @@migrate_richtext to migrate html-richtext to slate blocks or draftjs blocks.
    • Add @@migrate_to_volto to prepare existing sites for Volto.

Installation

Installation instructions are being added in this documentation pull request. You can see a preview.

If you cannot follow these docs in progress, the following older notes can be useful.

Some documentation about installation:

If you use Docker, we have some images:

  • plone/plone-backend (5.2 and 6.0)
  • plone/plone-frontend (Volto)
  • plone/plone-haproxy

If you don't do Docker, you will have to do the backend by hand.
The links above should give you information on how to install the prerequisites, like Python, also on Windows.
Here, we will focus on Unix-like systems (Linux, Mac OSX), but Windows should work as well.
The steps are:

  • Install the Plone (Classic) backend with buildout or pip.
  • Create the Plone Site in the browser.
  • Install the Plone frontend (Volto) with node.

More installation info at https://plone.org/download/releases/6.0.0a6

Plone 6

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Try Plone 6 Alpha 6

Download and installation:
https://plone.org/download/releases/6.0.0a6

Read more at the community forum:
https://community.plone.org/t/plone-6-0-0a6-released/15401

Plone Conference 2022 - Tickets for Sale

Posted by PLONE.ORG on June 30, 2022 08:45 PM

Plone Conference 2022 takes place in Namur, Belgium on October 10-16, 2022.

We have now opened ticket sales, with early bird prices!

The topics at the conference range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. Project management, process improvement, open-source, design, and community aspects are valued talk topics.

Get ready for the new Plone 6: the features, stability, scalability, and best-in-class security of an enterprise CMS, combined with a modern, easy-to-use, and powerful front end based on state-of-the-art web technologies.

Plone community is the most friendly open source community there is and always welcomes new people warmly.

Conference tickets include

  • Access to two days of training sessions
  • Access to three days conference
  • 40+ presentations
  • Interact with speakers, sponsors, and attendees
  • Join the community sprints

Get your tickets!

More info

If you have any questions, please contact [email protected], we are happy to help you!

Namur

Buschenschanksprint 2022 Report

Posted by PLONE.ORG on June 17, 2022 01:27 PM

This was the Buschenschanksprint 2022

stand up

After two years of working from home, sprinting remotely, and meeting via video conferences the Buschenschanksprint was for many the first occasion to see each other in person.

And meeting in person was amazing. We had many discussions, demos, and stand-ups which seemed just more effective than what we had over video conferences. Not that video conferences are ineffective, but it was high time to meet in person again.

The beautiful landscape of the "Südsteirische Weinstraße", the Buschenschankjause, and a glass of wine or two in the evening was also a fine thing to have.

We were at most 18 people directly at the sprint and two remote sprinters. The weather was fine, we could work outside and inside, on the terrace or in the arbor, depending on preference.

The space was big enough and the internet connection absolutely sufficient. We had some people in tents, 8 people in the "Ferienhaus List" and the rest at the Apartment "Sonnenberg".

For dinner, we went to nearby Buschenschenken, to a Restaurant, and did some BBQ ourselves.

This year Buschenschanksprint was a strategic sprint for the second time. We had a lot of plans for progressing Plone and related projects, and we did achieve a lot.

group pic

Progress during the sprint

Here is but a short list of headlines of sprint achievements.

To read the progress in full detail, check the community forum post: https://community.plone.org/t/this-was-the-buschenschanksprint-2022/15347

  • Barceloneta LTS improvements
  • Webpack Module Federation
  • Documentation and Training
  • Migrate to Volto
  • Work was done on an automatic migration path from Classic UI to Volto.
  • Relations in Volto
  • Mosaic
  • Datagridfield
  • Images
  • Mxdev/Mxmake
  • YAFOWIL
  • plone.api
  • Control Panel User Group Membership in Volto
  • Various other topics
  • Future of Classic Plone discussion

buschen3.jpeg

Participants of the Buschenschanksprint

  • Johannes Raggam (Organiser)
  • Alessandro Pisa
  • Manuel Reinhardt
  • Robert Niederreiter
  • Jens Klein
  • Maik Derstappen
  • Philipp Auersperg
  • Guido Stevens
  • Peter Mathis
  • Christine Baumgartner
  • Peter Holzer
  • Markus Hilbert
  • Alexander Pilz
  • Maurits van Rees
  • Philip Bauer
  • Georg Bernhard
  • Leo Auersperg
  • Katja Süss
  • Steve Piercy (Remote)
  • Fred van Dijk (Remote)

That was the Buschenschanksprint. We're already looking forward to the next one!

Plone Conference 2022 - Call for Papers Open!

Posted by PLONE.ORG on June 15, 2022 06:00 PM

Plone Conference 2022 takes place in Namur, Belgium on October 10-16, 2022. We have now opened call for papers, looking for all kinds of different talks related to Plone.

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. Project management, process improvement, open-source, design, and community aspects are valued talk topics. Plone 6 cases are of much interest this year.

There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences.

If you have any questions, please contact [email protected]. No matter if you are an experienced speaker or a beginner, we are happy to help you!

Submit your talk!

More info

ITONBOARD - IT community onboarding for young people

Posted by PLONE.ORG on May 30, 2022 01:46 PM

At the Plone conference in Ferrara 2019, after a talk by Christine Baumgartner and her two interns Ilvy and Nils, a bunch of companies discussed the future of on-boarding of young people into IT. Three companies, Abstract, Interaktiv and Klein & Partner, followed up on the idea and looked for funding to improve IT on-boarding with focus on free and open source software companies.

Erasmus-Logo.pngAfter further meetings we decided to apply for an Erasmus Plus project of the European Commissison. We expanded the scope from internships only to the whole process of onboarding. With Talentbrücke (Cologne) we found a partner in the field of the transition from school to job; with International Formation Center (Madrid) a company specialized in designing and managing training and development programs.

At the Plone Tagung in Dresden, very shortly before the first COVID19 lockdowns in Europe, we prepared the first application, but it failed. After adjusting our strategy to a broader community, and with remote work in focus, the second application of our project got accepted. In March 2021 we had our kick off!

logo-ITONBOARD-hires-plain.pngITONBOARD's goal is to introduce young students to career paths in open source web development. With that goal in mind we are developing five distinct products to guide students from a gamified experience to first-hand experience in schools; from e-learning opportunities to internship options. Our target groups are students, teachers and IT companies. The project is still ongoing. Some of our five outcomes are almost finished, others are still in development:
  1. an Escape Game for vocational orientation for students from 12 to 16 years old. Within the game, typical job situations and knowledge about IT professions can be experienced and acquired. Professions in web development as well as the concept of open source are presented to the players.
  2. a guide for teachers with a detailed concept for a project week at schools on the topic of IT career orientation. The central and associated learning content and materials were designed during the course of the project so far.
  3. online courses (MOOCs), which can be used as an alternative method of vocational orientation. The approach within e-learning formats is completely different from the classic and more theoretical information materials (flyers, texts, etc.).
  4. recommendations and guidelines for blended remote internships with short on-site phases in companies whose focus is on community-based, open source culture. Remote work and learning have had deep roots in the IT industry, and especially in the open source area, for decades.
  5. an internship exchange facilitating the arrangement of internships between interns and companies. For example, without the Plone Community the ITONBOARD project would never have happened. In the beginning the community was the initial backbone to discuss ideas.

We got in contact with other companies and collected their needs and experiences with interns, got lots of implicit information, personal feelings, and learned about failures and problems. In the Plone community we know each other and have many experts in different fields. Thus, many community members were keen to help with interviews, participating at workshops or reviewing material. Thanks to everyone donating their valuable time!

From the broader Plone community we interviewed Alexander Pilz (Syslab), Timo Stollenwerk (Kitconcept), Nejc Zupan (Niteo Web and project advisory board), Paul Roland (former president of the Plone Foundation and project advisory board) and Chrissy Wainwright (Sixfeetup and former president of the Plone Foundation) about their experience with internships and remote work. Furthermore we got valuable scientific insight into the field by our project advisory board member Univ.-Prof. Dr. Bernd Gössling (Uni Innsbruck). This gave us plenty of input for the blended remote internship guidelines and beyond!

20210825_143505(1).jpgFrom the community Katja Süss (Rohberg), Paul Roeland and Fred van Dijk (Zest Software) joined us at our multiplier event at the Schatzbergalm and provided valuable, decades long experience in remote work, teaching, diversity in IT and in general, and communities.

Additionally, Paul and Fred gave us a short interview. We also got support from other FOSS communities, here with a special mention of the OpenEDX community! And thanks to everyone we forgot to mention here, helping us with the project by sharing minds!

The project will continue for almost another year. We appreciate any feedback! If you want to learn more about ITONBOARD visit our website, send us an email or talk to Stefania, Christine, Jörg or Jens.

Plone Welcomes Students for Google Summer of Code 2022

Posted by PLONE.ORG on May 29, 2022 06:37 PM

For 2022, the Plone Foundation has been granted *six* Google Summer of Code student project slots.

Google Summer of Code is a global program focused on bringing more student developers into open source software development. Students work with an open source organization on a months-long programming project starting over the summer.

After careful selection of the many project proposals presented, we are pleased to announce that the following students will be working on various aspects of Plone:

Read more about their projects

To our new students: welcome to the Plone community – we wish you a great learning experience over the coming months. Congratulations!

Mentoring these students will be:

To our mentors: thank you for introducing new developers to open source and our community!

Beethoven Sprint Recap - Progress Towards Plone 6

Posted by PLONE.ORG on May 20, 2022 08:38 PM

The most effective (and fun) ways to move things forward in Open Source development are sprints - after years of remote sprinting due to COVID, we were finally able to hold a sprint in two locations simultaneously, with more than 25 people around the community gathering together.

The Beethoven Sprint was held in Bonn, Germany, and Bucharest, Romania.

Sprint Topics

These were the topics that we worked on:

Check the official sprint page for more details.

Bonn

Main goals were achieved, and we are now closer than ever to make Plone 6 a reality. The main blocker, Slate in core, is on its way. Other nice-to-have features to be included in Plone 6 include:

  • "base_view" default for custom view-less content types
  • initial iteration of the core Block Style Wrapper
  • extensive work on the new images srcset for frontends
  • initial design of the infrastructure to support them on the backend
  • rethought and refactored user and groups control panel
  • migrations, classic-to-Slate, draftJS-to-Slate, classic-to-Volto in general
  • DnD support for nested structures
  • Schema driven stock Blocks support
  • Containers for backend and frontend, nightly for buildout.coredev
  • and fixed tons of bugs... 

just to name the most important... the PR list is huge! 

Thanks to: Andrea Cecchi, Balazs Ree, Bernd Wolber, Érico Andrei, Fred van Dijk, Ion Lizarazu, Jakob Kahl, Jens Klein, Katja Süss, Maurits van Rees, Mikel Larreategi, Philip Bauer, Piero Nicolli, Rob Gietema, Steffen Ring, Thomas Kindermann, Timo Stollenwerk, and Victor Fernandez de Alba

Bonn

Bucharest

The Plone Beethoven 2022 sprint "two legs" summary from Romania:

  • The first Plone sprint ever organized in Romania!
  • Around 12 attendees in Bucharest and a few more remote from all over Romania
  • Thanks to Maik Derstappen for joining us so we can call it an international sprint 🙂
  • Facts: youngest Plone contributor for the sprint: 18 year old Ionuț Dobriceanu. Contributions merged!

Achievements:

  • volto-slate editor being merged into Plone6/Volto core advanced integration
  • "URL management"/"Aliases" control panel and contextual implemented for both backend and frontend
  • "Content rules" control panel and contextual implemented for the backend (API endpoints)
  • "Link Integrity for content objects" for Plone6 frontend implemented
  • ... and a couple of other generic issues solved!

Thanks to: Valentina Bălan, Laszlo Cseh, Maik Derstappen, Ionuț Dobriceanu, Valentin Dumitru, Krisztina Elekes, Alec Ghica, Andrei Grigore, David Ichim, Tiberiu Ichim, Răzvan Miu, Daniela Mormocea, Iulian Petcheși, and Alin Voinea.

Bucharest

Also, check out The Plone Newsroom podcast episode 8, where Philip, Fred and Victor talk about the Beethoven sprint, Plone 6 and Volto 16: https://plone.org/news/2022/the-plone-newsroom-podcast-episode-08

Upcoming sprints

Do you want to join the effort or meet new people? The friendly Plone community welcomes you to sprints this May:

5.2.8 Released

Posted by PLONE.ORG on May 14, 2022 08:15 PM

Some highlights of this release

  • Zope: Enhance cookie support. For details, see issue 1010
    For more changes see Change log — Zope documentation 4.6 documentation
  • waitress is updated to version 2.1.1 to mitigate a vulnerability in that package. As waitress no longer supports Python versions less than 3.7 it is not advised to run Plone 5.2 on Python 2.7 or 3.6 any longer, even though they are still supported by Plone itself. You get an older waitress version then. If you must use an old Python version, please switch to a different WSGI server. See the recommendations in the Zope documentation.
  • WARNING: the new waitress 2.1.1 does seem to suffer from a possible race condition leading to the process quitting. If you are affected by this, you can downgrade to 2.1.0 (which has a known security vulnerability, as mentioned above) or use a different WSGI server.
  • plone.app.linkintegrity: Track link integrity of referenced PDFs and other site objects in IFRAME SRC references.
  • plone.outputfilters: Resolve UIDs in SRC attribute of of SOURCE and IFRAME elements.
  • plone.app.querystring: Add lazy attribute to vocabularies to prevent fetching any results.
  • plone.schema: Use indent in json.dumps to make JSON readable in the widget.

Download Plone 5.2.8

The Plone Newsroom Podcast Episode #08

Posted by PLONE.ORG on May 12, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #08 on 12 May 2022 was all about Plone sprints - sprints in general and the recent Beethoven sprint in particular, which took place simultaneously in Bonn and Bucharest. It focused on the Volto 16 frontend that is coming in Plone 6. Philip, Fred and special guest Victor Fernandez de Alba discussed what got accomplished and what is planned after Volto 16. Go to the Newsroom page to view other episodes.

World Plone Day 2022 a Great Success

Posted by PLONE.ORG on April 29, 2022 10:15 PM

World Plone Day is an opportunity for members of the Plone community to celebrate and promote their favorite open source CMS. The 2022 event took place on April 27th and featured in-person events plus streaming and recorded content. There were:

  • 58 talks on a wide range of topics from beginner to advanced
  • 20 hours of video in 10 languages from 14 countries
  • 6 local events - in Italy, Brazil, India, Switzerland, and 2 in Romania

There were talks from long-time community members and from people who are relative newcomers. Here are a few highlights for a general audience:

Here are some highlights for developers:

And here are a few of the many talks in languages other than English:

You can find all the World Plone Day videos, and much more on the PloneCMS YouTube channel and the WPD 2022 playlist.

Progress on the New plone.org Built on Plone 6

Posted by PLONE.ORG on April 13, 2022 02:00 PM

The New Plone.org Will Be Plone 6

At the 2021 Plone Conference, the marketing team presented a plan was for how to improve and renew plone.org, the one and only place for Plone-related information:

The main ideas at the time were:

  • Plone.org will be migrated to Plone 6
  • Content needs refreshing, especially the frontpage and main landing pages
  • Visual theme will be out-of-the-box Plone 6 with Volto frontend, using available add-ons (many of which we expect will be added as core features)
  • Goal is to publish the new site by World Plone Day 2022 (April).

Great Progress at Sprints Plone org example

Since December 2022, we have organized 4 monthly sprints, working on:

  • Content and navigation structure
  • Other content work
  • Migration from Plone 5 to Plone 6
  • Visual design
  • Translation, default page content, add-ons
  • Plone 6 documentation
  • Volto improvements
  • Volto Blocks and installation
  • And more

We have updated the original plan to include a new visual theme and additional features that will eventually be available for others to use. A more ambitious scope has moved the release date somewhat further this year.

Special Thanks

The Plone Marketing and Communications Team is organizing the effort and working on the content overhaul, but we also want to express special thanks to:

  • Philip Bauer, from Starzel.de - Full content migration from Plone 5 to Plone 6, including the transition to Volto blocks
  • Érico Andrei, from Kitconcept - Foundation Member content type, installation, deployment, content type renewal, etc.
  • Steve Piercy and Katja Süss - Documentation effort resulting in new Plone 6 docs and organization around the work
  • Massimo Azzolini, Irene Capatti, Massimo Weigert, and Gianantonio Vecelli, from Giallocobalto - For massive content and structure redesign, visual design and layout, and UX design
  • Stefano Marchetti, Andrea Cecchi, Giulia Ghisini, and Andrea Baglioni, from Redturtle - For the new site's buildout which includes needed Volto blocks, curated add-ons, visual theme, and more: https://github.com/collective/volto-plone-org
  • Kim Nguyen, Brian Davis - For lessons learned from the previous renewal, content and structure discussion, theming mockups, etc. 
  • Victor Fernandez de Alba and Jakob Kahl - For Volto knowledge
  • And everyone else who has pitched in ideas, discussions, and questions!

This effort would not be possible without help from the awesome Plone community!

Want to Join the Effort?

The plone.org work is not done - our next sprint will be organized in May 2022. Feel free to join! There will be an event page, but meanwhile join the plone.org Discord channel for discussion. By the next sprint we hope to have a fully running site with migrated content available online for testing and content work.

If you have any questions or comments, please contact [email protected].

Stay tuned for more plone.org discussions and presentations at World Plone Day on April 27th 2022.

The Plone Newsroom Podcast Episode #07

Posted by PLONE.ORG on April 13, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #07 on 13 April 2022 featured news about the progress towards the Plone 6 Classic release, updates about Volto 15 development, and information about events (GSoC, World Plone Day, sprints (Zope, Beethoven, Buschenschank), and add-ons (collective.easyform, collective.honeypot and captcha). Go to the Newsroom page to view other episodes.

Plone 6.0.0a4 Released

Posted by PLONE.ORG on April 09, 2022 10:00 AM

Download and installation: https://plone.org/download/releases/6.0.0a4

The fourth alpha release of Plone 6 contains various fixes and improvements. Read more about the upcoming Plone 6.

Thank you to everyone involved!

Highlights

Changes since 6.0.0a3:

  • Use zc.buildout 3.0.0rc3 and setuptools 62.0.0 by default.
  • Update waitress to version 2.1.1 to mitigate a vulnerability in that package.
  • Zope 5.5.1: Enhance cookie support.
  • plone.staticresources: The big one: Updated JavaScript for Plone Classic, using ES6 modules. No more through-the-web compiling of JavaScript. See PLIP 3211.
  • Products.CMFPlone:
    • Remove RequireJS.
    • Remove default resource jQuery. It is added to the global namespace via the bundle.
    • Remove support for conditional comments in script and style tags. It's not supported since IE10.
    • Remove dependency on mockup. Mockup is now a npm package only and as such a dependency of plone.staticresources.
    • New resource registry to simplify CSS/JS registration.
    • Only "bundles" are registered - support of "resources" and "bundle resources" is removed.
    • Removed TTW compilation of bundles via r.js and less.js.
    • Property merge_with is no longer needed in HTTP/2 times and merging here unsupported.
    • Unique key for delivery is based on hash of bundle file, last_compilation property is deprecated.
    • PLIP #3279: Implement modern images scales. Add huge (1600px), great (1200px), larger (1000px), teaser (600px). Amend preview and mini (remove height constraint).
    • Add TinyMCE template plugin to the plugins vocabulary.
    • Add TinyMCE alignment classes, to avoid style usage.
  • plone.volto is now a dependency of the Plone package.
  • PLIP 2780: Move features of collective.dexteritytextindexer to core.
  • plone.app.dexterity:
    • Remove JavaScript from this package and move it to Mockup.
    • Modeleditor: Use pat-code-editor from Patternslib instead ACE. Make the model editing form usable without JavaScript. Allow editing the form even with XML errors to be able to fix the problem.
  • plone.recipe.zope2instance: by default do not create a temporary storage.
  • plone.scale: Removed deprecated factory argument from scale method.
  • plone.app.linkintegrity: Track link integrity of referenced PDFs and other site objects in IFRAME SRC references.
  • plone.outputfilters: Resolve UIDs in SRC= attribute of of SOURCE and IFRAME elements.
  • plone.app.querystring: Add lazy attribute to vocabularies to prevent fetching any results.
  • plone.app.theming:
    • Deactivate copy button and modal in theming control panel.
    • Remove all thememapper functionality from theming control panel, including Inspect/Modify theme and the Preview.
  • plone.app.users: Show unfiltered member fields for manager in user profile page.
  • plone.app.widgets:
    • Remove implicit dependency on Mockup. Mockup is no longer a Python package, only an npm package.
    • Update datetime pattern options for Patternslib pat-date-picker/pat-datetime-picker.
  • plone.autoform:
    • Fixes for latest z3c.form.
    • Reimplementation of ObjectSubForm and ISubformFactory, backported from older z3c.form.
  • plone.app.z3cform:
    • Use better types for inputs.
    • Use browser native date and datetime-local input together with patternslib date-picker.
    • Implement TimeWidget which renders <input type="time" />.
    • Use pat-validation in forms.
    • Fixed for latest z3c.form
  • plone.z3cform: compatibility with latest z3c.form.
  • plone.namedfile: Register AnnotationStorage as IImageScaleStorage multi adapter, both from plone.scale. Use this adapter in our scaling functions when we store or get an image scale.
  • Products.PlonePAS: Add separate GenericSetup profile to switch the Zope root /acl_usersto use a simple cookie login form. Useful when Zope root login and logout need to synchronize authentication state between multiple plugins, which is not possible with HTTP Basic authentication.
  • plone.app.layout:
    • Restructure global sections and searchbox markup for mobile navigation as offcanvas sidebar.
    • LiveSearch with support for images in search results.
  • plonetheme.barceloneta: sticky footer.

Note that changes may be mentioned only once, even when they involve multiple packages.

Plone 6

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Try Plone 6 Alpha 4

Download and installation:
https://plone.org/download/releases/6.0.0a4

Read more at the community forum:
https://community.plone.org/t/plone-6-0-0a4-released/15126

Volto 15.0.0 Released

Posted by PLONE.ORG on March 08, 2022 10:10 AM

The Plone community, in particular the Volto team, is happy to announce that Volto 15 is ready and shipped!

Volto is Plone's snappy, modern React front end powered by the RestAPI, and the default for Plone 6.

Volto 15.0.0

New feature highlights

Volto grid block

  • Add `cookiesExpire` value to config to control the cookie expiration @giuliaghisini
  • DatetimeWidget 'noPastDates' option: Take widgetOptions?.pattern_options?.noPastDates of backend schema into account. @ksuess
  • Add a new type of filter facet for the Search block. Heavily refactor some searchblock internals. @tiberiuichim
  • Add date range facet to the search block @robgietema
  • Introduce the new `BUILD_DIR` runtime environment variable to direct the build to run in a specific location, different than `build` folder. @sneridagh
  • Handle redirect permanent calls from the backend in the frontend (e.g. when changing the short name) @robgietema
  • Added id widget to manage short name @robgietema
  • Refactor language synchronizer. Remove it from the React tree, integrate it into the Api Redux middleware @sneridagh
  • Add blocks rendering in Event and NewsItem views (rel plone.volto#32) @nzambello @ksuess
  • Complete Basque translation @erral
  • Complete Spanish translation @erral

Breaking changes

  • Upgrade `react-cookie` to the latest version. @sneridagh @robgietema
  • Language Switcher no longer takes care of the change of the language on the Redux Store. This responsibility has been unified in the Api Redux middleware @sneridagh
  • Markup change in `LinkView` component.
  • Rename `core-sandbox` to `coresandbox` for sake of consistency @sneridagh
  • Extend the original intent and rename `RAZZLE_TESTING_ADDONS` to `ADDONS`. @sneridagh
  • Lazyload draftjs library. See the upgrade guide on how that impacts you, in case you have extended the rich text editor configuration @tiberiuichim @kreafox
  • Deprecating `lang` cookie in favor of Plone official one `I18N_LANGUAGE` @sneridagh
See https://6.dev-docs.plone.org/volto/upgrade-guide/index.html for more information about all the breaking changes.


Bugfix

  • Fix the `null` error in SelectAutoComplete Widget @iFlameing
  • Prevent the MultilingualRedirector to force 4 content load when switching the language @reebalazs
  • Fix the upload image in contents view @iFlameing
  • add "view" id to contact-form container for main content skiplink @ThomasKindermann
  • Fix loading indicator positioning on Login form submit @sneridagh
  • Fix redirect bug with URLs containing querystrings @robgietema
  • Fixed id widget translations @robgietema
  • Contents Rename Modal, use `id` Widget type @sneridagh
  • Fix overflow of very long file name in `FileWidget` @sneridagh
  • Fix overflowing issue in the toolbar @kreafox
  • Overwrite current block on insert new block. @robgietema
  • Fix hot reload on updates related to the config object because of `VersionOverview` component @sneridagh
  • Fix error when lock data is gone after an invariant error. @robgietema


Internal

  • Change prop `name` -> `componentName` in component `Component` @sneridagh
  • Add new RawMaterial Volto websites in production @nzambello
  • House cleanup, remove some unused files in the root @sneridagh
  • Move Webpack related files to `webpack-plugins` folder @sneridagh
  • Remove unused Dockerfiles @sneridagh
  • Update Docker compose to latest images and best practices @sneridagh
  • Improve flaky test in coresandbox search Cypress tests @sneridagh
  • Better implementation of the add-on load coming from the environment variable `ADDONS` @sneridagh
  • Turn `lazyLibraries` action into a thunk. Added a conditional if the library is loaded or in process to be loaded, do not try to load it again. This fixes the lag on load `draftjs` when having a lot of draftjs blocks. @sneridagh
  • Use `@root` alias instead of `~` in several module references. Most of the Volto project code no longer needs the root alias, so it makes sense to phase it out at some point @tiberiuichim
  • Alias `lodash` to `lodash-es`, as this will include only one copy of lodash in the bundle @tiberiuichim

Documentation

  • Upgrade Guide i18n: Make clear what's project, what add-on. @ksuess
  • (Experimental) Prepare documentation for MyST and importing into `plone/documentation@6-dev`. @stevepiercy
  • Fix broken links and redirects in documentation to be compatible with MyST. @stevepiercy
  • Update add-on internationalization. @ksuess
  • Add MyST and Sphinx basic configuration for rapid build and comparison against MkDocs builds. @stevepiercy
  • Fix many MyST and Sphinx warnings. @stevepiercy
  • Remove MkDocs configuration. See https://github.com/plone/volto/issues/3042 @stevepiercy
  • Add Plone docs to Intersphinx and fix broken link. @stevepiercy
  • Get version from `package.json` @sneridagh
  • Remove legacy folder in docs @sneridagh
  • Backport docs of RAZZLE_TESTING_ADDONS environment variables. See https://github.com/plone/volto/pull/3067/files#diff-00609ed769cd40cf3bc3d6fcc4431b714cb37c73cedaaea18fe9fc4c1c589597 @stevepiercy
  • Add missing developer-guidelines/typescript to toctree @stevepiercy
  • Add Netlify for a preview of Sphinx builds for pull requests against `master` and `plone6-docs`. @stevepiercy
  • Clean up toctree errors by removing obsolete files, adding `:orphan:` field list, and reorganizing some files. @sneridagh and @stevepiercy
  • Switch to using netlify.toml to configure Netlify Python environment. @stevepiercy
  • Convert admonition syntax from Markdown to MyST. @sneridagh
  • Make links build both in Volto and Plone documentation. See https://github.com/plone/volto/pull/3094 @stevepiercy
  • Fix StoryBook build @sneridagh

What's Next

Where do we go from here? Plone 6! Right now, the only major feature missing is content rules and the new Slate editor. The rest of Plone’s features are covered in Volto 16.

So the work is not over yet. We still need helping hands and contributors to continue the effort to make Plone 6 a reality. Everybody is welcome!

Thanks!

We would like to thank all the people involved in creating Volto 15. It is amazing how much we were able to accomplish as a team, and as a community, during the last months.

See https://6.dev-docs.plone.org/volto/upgrade-guide/index.html for more information about all the breaking changes.

Try Plone 6 today

Feel free to try out Plone 6 with Volto 15:

Plone selected for 2022 Google Summer of Code

Posted by PLONE.ORG on March 07, 2022 08:41 PM

Once again this year members of the Plone open source community will mentor students and other young people interested in learning how to develop their skill and to contribute to an open source project. Plone has benefited greatly from the contributions of GSOC students since it began participating in the program in 2006. Many of those GSOC students have gone on to become core contributors to Plone as well as productive members of other open source software communities.

Students whose applications are accepted will work with a senior Plone programmer on one of a list of projects with the goal of making a contribution to the Plone code base or a related project. The Plone Foundation supports successful GSOC students by providing support for attending the annual Plone conference and for meeting other members of the diverse Plone / Python / Zope community.

Visit the GSOC home page and learn how to apply for the program

Check out our list of potential student projects for Plone in 2022

Student applications will open on April 4, 2022.

Join us !

Programmers report out at sprint in Sorrento, Italy

The Plone Newsroom Podcast Episode #06

Posted by PLONE.ORG on February 28, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #06 on 28 February 2022 featured news about new Foundation members and the 2022 Plone Conference dates, a discussion of the plone.org relaunch work and issues encountered, Volto, Classic and collective.exportimport development news, and a follow up to the composite page builders discussion in episode #05. Plus Philip and Fred issued a call for Volto translators and a call for World Plone Day talks. Go to the Newsroom page to view other episodes.

Welcome to Three New Members of the Plone Foundation

Posted by PLONE.ORG on February 23, 2022 10:25 PM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on February 17, 2022.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Giulia Ghisini turning world upside down

Giulia Ghisini

Giulia is a frontend developer who works on the Volto development team. She has sprinted at Sorrento once and has participated in Plone conferences since 2019.Giulia works at Red Turtle and lives in Ferrara, Italy.

Piero NicolliPiero Nicolli

Piero also works with Red Turtle in Ferrara. He has been using Plone since 2015 and has made occasional contributions since then. Piero's focus has shifted to Volto after beginning with contributions on plonetheme.barceloneta and plone.staticresources. He is currently part of the Volto Team.

Piero attended Plone Conferences 2018, 2019 and 2021, and he helped organize the 2019 Ferrara conference. In addition to several sprints, Piero gave a talk about frontend development at PloneConf 2019 and about a use case for a repeatable Volto themes at PloneConf 2021. He is active on Discord from time to time and answer questions there when available.

Alessandro Pisa at SorrentoAlessandro Pisa

Alessandro works remotely for Syslab from Ferrara (Italy) and has been
a Plone user and developer since 2008. He has been a Framework Team member since 2017. He is a Plone core developer and contributes and maintains some packages in the Plone ecosystem.

Alessandro is a long-time contributor to the community and helps to manage the collective organization on GitHub. He has attended many sprints and makes sure to reach out to newcomers when conference time comes around.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Custom Volto configuration to fix Babel problems with react-leaflet

Posted by PloneExpanse on February 06, 2022 04:05 PM
I’ve started working on a new Leaflet-powered Volto map block and the first thing that happened while loading react-leaftlet was an error reported by the browser: Module parse failed: Unexpected token (10:41) in @react-leaflet/core/esm/path.js ... const options = props.pathOptions ?? {}; ... The problem is that is, for some reasons, the transpiled JS bundle includes code using the nulish coalescing operator This is already a problem reported in react-leaflet and it happens because the distributed transpiled library includes that code.

Questions for the Feburary Steering Circle?

Posted by PLONE.ORG on February 02, 2022 06:05 PM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be Feburary 15th at 14:00 UTC (15:00 CEST). Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone 6.0.0a3 and Plone 5.2.7 Released

Posted by PLONE.ORG on January 31, 2022 07:50 PM

Important security fix applied

See forum post for more details and easy workaround:
https://community.plone.org/t/security-fix-for-image-view-fullscreen-cache-poisoning/14757

See also CMFPlone security advisory on GitHub.

Plone is vulnerable to reflected cross site scripting and open redirect when an attacker can get a compromised version of the image_view_fullscreen page in a cache, for example in Varnish. The technique is known as cache poisoning.
Any later visitor can get redirected when clicking on a link on this page. Usually, only anonymous users are affected, but this depends on your cache settings.

All Plone versions are vulnerable. It depends on your Plone version and the Image content type which package is vulnerable: Products.CMFPlone, plone.app.contenttypes or Products.ATContentTypes.

The Plone Security Team has released fixes for Plone 5.2:

  • plone.app.contenttypes 2.2.3 (see advisory)
  • Products.ATContentTypes 3.0.6 (see advisory. If you are on Python 3 you will not be using this package.

and for Plone 6:

  • plone.app.contenttypes 3.0.0a9 (see advisory)

Today, Plone 5.2.7 and 6.0.0a3 have been released with these updated packages. Separate announcements will follow.

If you have any questions or comments about this advisory, email us at [email protected]. This is also the correct address to use when you want to report a possible vulnerability. See our security report policy.

Highlights of the Plone 6 Alpha 3 Release

https://plone.org/download/releases/6.0.0a3

Changes since 6.0.0a2:

  • plone.app.contenttypes: Security fix: prevent cache poisoning with the Referer header.
    See security advisory.
  • Updated the versions of the build requirements: setuptools to 59.6.0, zc.buildout to 3.0.0rc1, pip to 21.3.1.
  • Zope 5.4:
    • Add support for Python 3.10 (Plone does not have this yet).
    • WebDAV fixes.
    • https://zope.dev is now the canonical Zope developer community site.
  • plone.volto: Removed collective.folderishtypes dependency.
  • Products.CMFEditions:
    • Got rid of the skins directory. Most items in here have been moved to browser views. Some were no longer used, or had an alternative, and were removed.
    • The VersionView class is deprecated because it contained just one method that is now part of the @@plone view.
  • plone.app.linkintegrity: Track integrity of video and audio files in HTML source tags.
  • plone.app.uuid: Speed up uuidToPhysicalPath and uuidToObject.
  • plone.namedfile:
    • Make DefaultImageScalingFactory more flexible, with methods you can override.
    • Drop support for Python 2.7. Main target is now Plone 6, but we try to keep it running on Plone 5.2 with Python 3.
  • diazo: Removed FormEncode test dependency.
  • Pillow updated to 9.0.0
  • plone.app.content: Deprecate the human_readable_size method of the ContentStatusHistoryView class because the one from the @@plone view should be used.
  • plone.app.layout: Improved the Global section viewlet:
    • Catalog based navigation.
    • Allow more customization by adding methods as hooks.
    • Various performance optimizations.
    • Deprecate now unused navtree_depth property.
  • plone.app.layout: Removed deprecated methods.
  • plone.app.layout: Add viewlet to display customizable favicon. See the Site Settings.
  • Various packages: No longer use deprecated property types ulines, utext, utoken, and ustring, but their non-unicode variants, without a u at the beginning. See issue 3305.
  • plone.restapi:
    • Enhance @addons endpoint to return a list of upgradeable addons.
    • Add support for DX Plone Site root in Plone 6. Remove blocks behavior hack for site root in Plone 6.
  • Products.CMFPlacefulWorkflow: Removed the CMFPlacefulWorkflow skin layer.

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Plone 5.2.7 released

Specific release notes for Plone 5.2.7:

Some highlights of this release are:

  • `plone.app.contenttypes` and `Products.ATContentTypes`: Security fix: prevent cache poisoning with the Referer header. See security advisory.
  • `plone.app.linkintegrity`: Track integrity of video and audio files in HTML source tags.
  • `plone.app.z3cform` and `plone.app.textfield`: Enable multiple wysiwyg editors (use default editor registry setting).
  • `plone.namedfile`: Make `DefaultImageScalingFactory` more flexible, with methods you can override.
  • `plone.app.layout`: Improved the Global section viewlet:
    • Catalog based navigation.
    • Allow more customization by adding methods as hooks.
    • Various performance optimizations.
    • Deprecate now unused navtree_depth property.
  • `diazo`: Removed `FormEncode` test dependency.
  • `plone.restapi`: Be permissive when testing the schema of the querystring endpoint.

https://plone.org/download/releases/5.2.7

On The Road to Plone 6 - Volto 14 Released

Posted by kitconcept GmbH on December 22, 2021 09:24 PM

Volto continues to innovate at a fast pace towards Plone 6. Today we released another major milestone on our road to Plone 6: Volto 14.

Volto 14 was in the making since September 2021 and it is in active use in various projects at Eau de Web, Red Turtle, Rohberg, kitconcept, and others.

Volto 14 comes with a set of new exiting features: a new search block that supports faceted search, locking support, a new seamless mode that makes deploying Volto easier, a new mobile navigation and support for Node 16.

Faceted Search Block

The new search block allows editor to create sophisticated faceted searches through the web without writing a single line of code.

Editors can define criteria for the content that is listed, like in the existing listing block in Volto.

Then editors can then choose arbitrary facets that are displayed to the users to choose from to narrow down the search.

Locking

Locking in a Content Management System is a mechanism to prevent users from accidentially overriding each others changes.

When a user edits a content object in Plone, the object is locked until the user hits the save or cancel button. If a second user tries to edit the object at the same time, she will see a message that this object is locked.

volto14 locking Content object in Volto 14 that is locked by another user

Locking requires at least plone.restapi 8.9.0 or plone.restapi 7.4.0 to be installed.

Seamless Mode

The new “seamless mode” allows zero configuration deployment by avoiding hardcoded environment variables in builds involved, and establishing good sensible defaults when setting up deployments (and also in development). So the developer/devOps doesn’t have to overthink their setups.

These are its main features:

  • Runtime environment variables
  • Unified traversal ++api++
  • Use Host header to auto-configure the API_PATH

and these immediate benefits:

  • Avoid having to expose and publish the classic UI if you don’t really need it
  • If possible, avoid having to rewrite all API responses, since it returns paths that do not correspond to the original object handled and “seen” from Volto, so you have to adjust them (via a code helper) in a lot of call responses.
  • Simplify Docker builds, making all the configuration via the runtime environment variables

Seamless Mode requires at least plone.rest 2.0.0a1 to be installed.

New Mobile Navigation

We polished the mobile navigation for Volto 14 to improve the user experience on mobile.

New mobile navigation in action on the Plone Conference 2021 website

Node 16

Volto 16 will support Node 16 in addition to Node 14 and Node 12. Node 16 will receive security updates until April 2024.

node 16 Node 16 release schedule from nodejs.org

Plone 6

Volto 14 is an important step towards Plone 6. Volto 15 will switch the default editor from DraftJS to Slate editor and it is planned to be ready in Q1/Q2 2022. This is the last big step for Volto before Plone 6 can be released.

David Ichim: What's new in Plone 6 frontend for developers

Posted by Maurits van Rees on November 08, 2021 11:18 AM

Distilled from the latest work done in Volto, we're showcasing some patterns, features, or enhancements that have landed in Volto from the last year to the present. We will also have a glimpse of what is ahead in the future of Volto with the new features roadmap.

In the past year we had four major releases, 40 minor releases, 36 alpha releases, 25 patch releases, for a total of 105 releases. Plus some new tooling and tool releases, like plone i18n and plone generator.

New Volto config, dubbed as Volto's Configuration Registry, introduced to fix circular import dependency problems. Hot Module Reloader was fixed. Read the upgrade guide.

New i18n (internationalisation) infrastructure. This is now a separate package. Same is used for generation of i18n in add-ons.Read the upgrade guide.

Forms as schema. Forms should be constructed from schemas instead of by hand. InlineForm allows us to create forms for blocks from a schema. Blocks can have variations, or we can extend it. Read the edit component documentation.

New widgets:

  • Object List Widget. Similar to the original DataGridField. Used in core by the Search Block facets.
  • Object Browser Widget is now a separate widget, instead of part of a block, and now allows the addition of external content.
  • Querystring Widget. Behaves like its counterpart from plone.app.querystring. Allows to create search criteria, used by the search block.
  • URL Widget. Used on text inputs, it knows to validate their value as a url, both internal and external.
  • Vocabulary Terms Widget for a JSONField, acts as a source for a SimpleVocabulary or Choice field. Play with it in the storybook.

New core blocks:

  • Search block
  • Teaser block (ongoing work).

Pluggable framework, similar to React's Portal component. See the talk by Tiberiu and see the Pluggables development recipe.

Storybook provides a sandbox to build and test visual components in isolation. Currently it is only setup to be used by Volto core. We need help with work to have storybook setup with adding. See the Storybook talk held by Victor, and see the storybook itself.

Critical CSS: inline the critical CSS for improved performance. Run critical-cli to output critical.css. This is then inlined in the headers, while regular CSS is moved to the bottom of the body. See the deployment documentation.

Lazy Loading utilities. Introduced injectLazyLibs HOC wrapper to inject lazy-loaded libraries as properties to your components. These components are only loaded once all your main libs are loaded.

Express.js middleware. Volto uses this for SSR and static resources serving. You can now write custom middleware and add it to settings.expressMiddleware.

API expanders allow the expansion of different API endpoints from Volto with calls from your custom endpoints. Avoid adding too many expanders if they are not critical to the initial page.

External routes: useful when another application is published under the same top domain as Volto. If Volto would fetch the content for that path, it will break. You can disable this path in config.settings.externalRoutes. You can also use regular expressions.

Seamless mode, introduced in Volto 13, enhanced in Volto 14, which has already seen a lot of alpha releases. Originally, we tried to unify both frontend and backend servers under the same path, but this was tricky, causing various problems. We settled on a new ++api++ traversal for getting information from the backend. Also, to come closer to zero config, you can now pass environment variables at runtime instead of build time. This means you can generate one build, and use this in all environments (testing, production). Read the deployment documentation.

Context navigation component. This is a navigation portlet component, similar to Classic Plone. The view is there, but you need to enable it. See the development recipe.

There is some work in progress:

  • Slots are Volto's answer to portlets, see the Volto Slots talk by Tiberiu Ichim for more details.
  • Image proxy: image scale generation done by a middleware instead of plone.scale.
  • Authentication from backend.
  • Replace Draft.js editor with Volto Slate. What is missing is a migration tool from one to the other. But work has started on a block conversion tool.
  • Async blocks that work with SSR.
  • Defaults in blocks form.

Future work:

  • Defaults in all widgets
  • Enable blocks enhancers in all blocks
  • Storybook in add-ons
  • Use newest react-intl package
  • Refactor the folder contents component
  • Form editing text enhancements, making it easier to modify text inputs.
  • A "group block" included with Volto
  • Quanta toolbar

I did nothing, I just brag about what others have done. So thank you Volto early adopter community!

Tiberiu Ichim: Volto Pluggables

Posted by Maurits van Rees on November 08, 2021 09:55 AM

This presentation is an introduction to the new Volto developer-targeted feature, the Pluggables framework. It is more an argument for extensibility in CMS UI and in Volto.

Basically, with Pluggable a central component provides a pluggable slot that other components can fill, like this:

<div className="toolbar">
  <Pluggable name="toolbar-main" />
</div>
// ...
<Plug id="quicklinks" pluggable="toobar-main">
  <Button />
</Plug>

This is a Volto port of https://github.com/robik/react-view-slot.

The big picture:

  • I work with Eaudeweb Romania
  • Our client EEA is an early adopter of Volto
  • The strategy is: move everything to Volto
  • But the EEA sites are less brochure, the CMS side is really strong.
  • We build powerful UIs for power users.
  • The EEA already has 91 Volto repositories on GitHub. How can we scale that? Can we write an add-on to make it easier to write an add-on?

In React, data flow is top to bottom. A parent component passes properties to children and children communicate with the parent by emitting events. This makes sense and works well. For "out of tree" data you need Redux. There is no protocol for add-hoc communication between components.

UI state is fluid. Extensibility means reusability and scalability. This is hard. You need to design upfront. Plone backend uses the Zope Component Architecture, which means pluggability is baked in, it is very natural. You can view Pluggables as viewlets-on-demand, but they are really not. But yes, you can think about a Pluggable as a viewletmanager and a Plug as a viewlet.

You can overwrite a Plug with a Plug, by registering it with the same id. So if the original Plug gives you color blue, you can overwrite it with color red.
You can do custom rendering of Plugs within your Pluggable, by iterating over all Plugs and for example wrapping each in a div with a class name.

Showcase: volto-workflow-progress (main toolbar plugin) and volto-editing-progress ("sidebar" for plugin).

Limitations:

  • No Server Side Rendering
  • Watch out for dependency lists.
  • Limited adoption for now.

Implementation in pseudocode:

First the context:

<PluggablesProvider>
  <Pluggable name="top" />
  <Plug name="delete" />
</PluggablesProvider>

Then the dumb version of the Plug:

const Plug = ({id, children}) => {
  const { register } = useContext(PluggablesProvider.Context);

  React.useEffect(() => {
    register(id, () => children);
  });

  return null;
}

Takes a bit of study, but in the end it is not so hard.
The Pluggable becomes a bit simpler:

const Pluggable = (name) => {
  const { getPlugs } => useContext(PluggablesProvider.Context);
  return getPlugs(name).map(f => f());
}

Site updated from Plone 2.5 to Plone 6.0

Posted by Maurits van Rees on November 04, 2021 11:34 AM

I am one of the Plone Release Managers, and have been working on Plone 6, which is now in alpha stage. But my personal website was still using the ancient Plone 2.5:

Screenshot maurits.vanrees.org Plone 2.5 Screenshot of my website end of October 2021: still Plone 2.5

Often I have made plans to update my site to:

  • Plone 3
  • Grok
  • Plone 4
  • Plone 5
  • Plone 5.2
  • finally Plone 6

Long ago it was clear to me that an inline migration would not be practical. It would take too many steps: update the code to Plone 3.3, migrate the data, to Plone 4.3, migrate data, to Plone 5.2 Python 2.7, migrate data, Plone 5.2 Python 3, migrate data, Plone 6, migrate data.

Additionally, the question was how to handle the weblog, which is the main content. This was using Products.Quills, a Plone add-on for blogs. Updating to Plone 3 could have worked at the time, but this was made harder by some some custom code I added. This enabled me to use it as a podcast. I used this to enrich some of my summaries of sermons from my church with the actual audio of the sermon. I doubted whether to even include this content in Plone 6, as the last sermon was from 2008. I hate breaking links, so I kept it, although a bit hidden.

Another point that needed some extra attention, was that most, if not all, blog entries were not written in html, but in ReStructuredText. I make a lot of summaries of talks when there is a Plone Conference. The html editor on Plone 2.5 did not work anymore, or I disabled it to a simple textarea. I always open up the same text editor that I use for programming (previously Emacs, currently VSCode), and type the summary there. I much prefer writing ReStructuredText there, especially when I simply need text without any markup. I then paste it in Plone, without fear of losing all my work when my internet connection dies.

Lastly, I have an RSS/atom feed which is used by planet.plone.org and maybe others to stay updated when I add a new blog entry. I did not want this url to change.

Anyway, about six years ago I decided that I would use collective.jsonify to export my site, and then import it using transmogrifier. But time passed without any progress. This year, collective.exportimport was shaping up to be the preferred way to import the data. For export you can use it in Plone 4.3 as well, but definitely not in Plone 2.5.

At the beginning of this week I looked at jsonify. Didn't I have a local copy of my website on my laptop, with collective.jsonify installed? No! And it was not installed on the live site either. And installation would be hard, because the site uses Python 2.4, and currently you cannot even reach PyPI with older versions of Python 2.7.

Mildly panicked, I started on a custom script to export the content, still as json. Some details aside, this actually was not so hard. I looked at what collective.exportimport expected, and created the Python list of dictionaries accordingly. Then do a simple json.dumps() call and you are done. Except that this gave an ImportError: the json module is not available in Python 2.4. Okay, install simplejson package. But you need PyPI access for that, which does not work. Workaround:

  • Manually download an egg of Python 2.4-compatible simplejson 1.7 and save it in the buildout directory.
  • cp bin/instance bin/instance-json
  • Edit the new script and add the simplejson egg to the system path.
  • bin/instance-json run export_mvrsite25.py

After that, it was not too hard anymore. I used plonecli to create a new add-on with Plone 6.0.0a1. I actually do not yet use the add-on code, except for loading a minor patch that I added, but this gave a reasonable, modern Plone 6 buildout. Create a Classic Plone site, tweak a few settings (let Folder use folder_workflow, allow English and Dutch, navigation depth 1, enable syndication, configure caching and mail), import the content with collective.exportimport, edit a bit, and done.

The weblog now consists of standard Folders and Pages. To improve the view, I added a Collection, showing the latest pages first, and showing all content of the last seven blogs. I enabled syndication here, so it has an RSS/atom feed.

The weblog has always advertised two atom feeds:

  1. One for all weblog entries, at https://maurits.vanrees.org/weblog/atom.xml
  2. One for weblog entries with keyword 'plone' at https://maurits.vanrees.org/weblog/topics/plone/@@atom.xml

In the new site, the first one kind-of worked out of the box, but it only showed the items that were directly in the weblog folder, and this is not where my weblog entries are. I solved this with a patch to Products.CMFPlone.browser.syndication.views.FeedView: when atom.xml is viewed on a folder, check if it has a default page, and show the atom.xml of this default page instead. In this case, the default page is a Collection with the correct settings. So the general feed will keep working.

For the second one, my first idea was to create a folder 'topics' and within it a Collection with id 'plone'. Problem: 'plone' is a reserved word and cannot be used as id of a content item, so the id became 'plone-1'. Solution here: create the 'plone-1' collection directly in the weblog folder, and do a redirect in the frontend server (nginx):

 rewrite ^/weblog/topics/plone/@@atom.xml /weblog/plone-1/atom.xml last;

And that's it! My website is now on Plone 6.0.0a1:

Screenshot maurits.vanrees.org Plone 6.0 Screenshot of Site overview in new site.

There are some more details that I could go into, like splitting up the buildout into multiple parts, with tox as main way to build everything, in preparation for moving more and more parts to pip-only. But that will have to be for another time.

Meanwhile: have a look around, and enjoy the fresh look!

Steering Circle

Posted by Maurits van Rees on October 31, 2021 02:39 PM

Volto 14 alpha 23 is out. So still in alpha, but companies are using it in production. Should be final soon. Some plans for Volto 15. Created plone.volto integration package, where we try to give an easy transition from earlier company-specific versions. plone.restapi as always is pretty boring, stable. Erico worked on Docker integration.

Plone 6 alpha 1 is out. Eric sent an email for some coordination, like docs, training, accessibility, installers. If you want to be involved, let me know.

Franco has stepped out of the Framework Team, thank you for all your work. There is discussion about the role of the Framework Team. Plan is to keep it running, some more people have been asked.

Membership team: we have some people in sight as new members. Erico is stepping down as team lead, Victor is stepping up.

Security: Plone 6 will have 5 years security support. Synchronizing with Zope team. Some new members may be coming.

Marketing: busy with conference, also after the conference. Busy with plone.org renewal.

Installers: see the talk by Jens Klein earlier. Plone 6: no more installer, but we do have tooling. There are Docker images. We may want to reduce the role of buildout, and focus more on pip.

Plone Conference: you are looking at it. Some tasks to do afterwards. If anyone is interested in getting a certificate for following a training, ask us, and we can send it.

Internationalization: new branches for Plone 6, so Plone 5 uses different branch. New releases for 5 and 6. Updating po files, looking at i18n in Mockup.

Admin/Intrastructure: servers are still running. Cat herding sysadmins for doing stuff, keeping things up to date.

Trainings: relaunch is complete. We have three new trainings: Theming Plone Classic, Deploying Plone 6, Getting Started with Plone 6 (that one only in video). Various have seen major updates. Need to work on landing pages (we have two), remove the number 5 from the url, update some more trainings. Maybe Mastering Plone Classic training, but hard with navigation due to duplicate section targets when copying. Migration training would be good. We need to prune and tag some.

Plone Classic: We did polishing on Barcelonate, it is pretty ready. Focussing on bobtemplates before the trainings, making theming easier. JavaScript/ES6 is the remaining big issue. Plan is to finish it this year, we are quite far. We need other people helping us out.

Documentation: will be releasing a new Plone 5 branch today. For the new stuff, the tech stack is ready. New version of automated screen shot is about ready. We don't want a duplicate of the training, but we can automatically include code of it, so there is only one source of truth. The style guide is not always followed, seeing what we can do about that. Biggest point is missing documentation. There are now branches where the various teams can add and edit their content. We may change things, but we take this as input for the final structure.

Fred van Dijk: 7 things that can surprise you when you start customising or developing for Plone

Posted by Maurits van Rees on October 29, 2021 09:06 AM

This is a basic rundown and summary of our beloved subjects like ZODB persistence. Traversal. The view/viewlet/portlet trinity. How is a call handled in Plone. The differences between zcml and generic setup. Utilities and the ZCA. restrictedTraverse. The Plone catalog.

These are surprises that I have encountered myself, or that I have seen on faces of people I have trained or worked with during the years.

  1. Everything can or could be done through the web (TTW).

Zope vision from the nineties. Why don't we use this dynamic language called Python so we can change things TTW? It's so easy.

  1. The learning curve.

It starts easy, but then you hit what we call a Z-shaped learning curve. Dynamic Python makes things easy, and then it makes things hard, at least when you put Zope on it, then CMF, then Plone. Plone the product on top of a CMS on top of a framework on top of a language. We have a product, a framework, a stack, so it is hard.

  1. Five levels of conceptual complexity.

It helps to teach all the levels. Give new users a drawer for each level, so they have a box that they can put some info in.

You have:

  • browser: html, css, js
  • frontend: transforms, templates, zpt, metal, diazo
  • application logic: views, viewlets, portlets, adapters, Zope Component Architecture
  • dynamism: GenericSetup, zcml, xml, zope schema
  • programming language: Python, buildout, pip
  • package WorkManager (OS): apt, rpm

4 Same language/formats on different levels:

  • XML is used for the ZCA, GS, zope.schema, Diazo rules
  • package manager: buildout, pip, setuptools, GS

But: there is no magic. It is just advanced technology.

Startup:

  • Python uses sys.path modules to start bin/instance
  • ZCA loads site.zcml, package includes, other zcml to change configuration.
  • Then we have a runtime environment with objects and classes, the ZODB. GenericSetup is then some XML that you can use to change the ZODB.

So the ZCA overrides components in runtime. The alternative is to edit core files, maybe compile them, and restart. Much less nice and not sustainable.

So now we have a Plone process running.

  1. Zope is not that complicated.

Over HTTP Zope gets a request, does traversal, finds an object in the ZODB, loads it in memory. Then on top of this object we show a browser view / template. The template hooks into a main template, maybe does some sub requests, some Diazo, and we end up with some html and we are done.

This is all still 'lies to children'. It is simplified until we are able to understand more. With increment exposure to these concepts, it will stick more. It is complicated, but there is no magic.

  1. Acquisition.

It is traversal in the wrong direction.

  1. When you try to explain things, you improve your own understanding.

There is so much Plone content online: training, docs, youtube, github, discourse. We all learn in different ways, with own preferences, reading, viewing. There are so many repositories on github that you can explore for new ideas. Just yesterday Philip did a talk about exportimport and afterwards I did it, but from a different angle. It helps.

The community is our biggest asset.

Annette Lewis and Will Gwin: From Zope to Plone: Thinking User-First During Migration

Posted by Maurits van Rees on October 28, 2021 02:39 PM

Migrating a site is always a challenging task, but when you have dozens of subsites with specific brand standards and custom user functionality, the challenge becomes mammoth. Six Feet Up worked hand-in-hand with Purdue's College of Engineering to migrate their existing Zope site and its subsites into a new Plone instance running on Plone 5.2 / Python 3. Throughout the migration process, we considered the project scale, timelines, and limiting the impact on end users, all while managing the balance between user needs and best practices. During this presentation, you will learn:

  • why it matters to think user-first during migration,
  • about creative solutions for translating content and functionality into Plone, and
  • how to successfully migrate subsites.

An overview of the project:

  • HigherEducation always seems to go a bit slower, certainly with migrations.
  • Our previous CMS was built in Zope and was getting extremely old.
  • We have been using Python since 2001.
  • There were security concerns and modernization issues.

The impact:

  • Only 15+ content editors.
  • 40+ public facing subsites
  • 30,000 total pages.
  • 20+ departments and units

Why did we select Plone as the next CMS?

  • It is a modern CMS solution
  • Python-based, so that fit what we currently have.
  • It is built on top of the Zope web framework that we were already using.
  • We looked at Drupal, Wordpress, and more, but that would have been a too big undertaking.

This was in 2018.

Laying out the solution. From requirements to action. Who are our users and what do they want? Not just our direct client (Will and his team) but their clients/users.

Challenges during development:

  • Purdue University changed its brand look. We had to seamlessly blend subsites into the existing parent site.
  • Convert all content types and templates from Zope to Plone.
  • Keep sight of the user experience in both environments. Could they use the new environment without too much training, or needing to have too much tech knowledge?

Determine the project essence. Distill the requirements down into broad categories: accessibility, usability, flexibility, security.

What is the path to successful collaboration?

  • The absolute best might not be the right answer.
  • It's okay to say no to an idea, but you dhouls have an alternative ready.
  • Aim for the best, avoid the dangerous, end up somewhere in the middle sometimes.

On to our development goals:

  • Do things in a Plone way. Plone uses Zope, but Zope may do some things in a different way than is the best way in Plone.
  • So observe best practices.
  • Make it intuitive and keep it familiar for the editors.

Solutions at a glance

Migrating site content:

  • We wanted to move subsites one by one, as needed.
  • Translate existin content to Plone content types
  • We could re-import content over existing content non-destructively.

Theming: retrofitting Plone into an existing theme. That is what Diazo was made for, bridging the gap between Plone and the theme, especially since the theme is 'living', with subtle changes coming in often.

Each subsite had a browser view named local.css to change some things. Not really what you want in Plone, but they really needed it, as a way to make subsites or sections look different. So we added an action to edit the local css, inheriting from parent folders.

We created a subsite settings control panel. We used lineage.registry for this. All kinds of customizations can be done based on that, for example add extra text and links in the navigation menu. They used to be able to do this in Zope as well, but that was with various properties, and much more code oriented.

We use Mosaic for flexibility of layout. In Zope we had blocks for layout. Mosaic took this a step further into a nice UI with drag and drop. It gives faster site prototyping and development.

The sandbox: we wanted to have safe spaces for content experimentation. Completely separate from production environment. It is used for new user training and testing. It is quick and easy to reset.

This migration project has been a constant collaboration between the Purdue communications office, Engineering Computer Network, and Plone company Six Feet Up. The content editors feel empowered to make complex changes, without constant oversight from my team.

Alan Runyan: Building a Secure Cross Platform Mobile/Tablet Application (Flutter) using Plone as Backend Server

Posted by Maurits van Rees on October 28, 2021 01:44 PM

Enfold has been working on a secure cross platform mobile application the past eight months. Walk through of the Requirements, Security, Flutter framework, Backend configuration of Plone, Authentication and Lessons Learned. Our goal is to have a free public release of a limited version of the application Q4 2021.

It was an adventure for us. The core team never built a large mobile application. We did not know what we did not know.

The big picture:

  • A mix of devices (Android, iOS) needs to synchronize files to and from Plone.
  • All services are self-hosted in GovCloud. So we have no central database or server that we control.
  • If this becomes a success, then future phases may require a lot of certification of the codebase, code reviews.
  • Initially we worked with 15k devices, supporting 40k would be a success, the ceiling that we might support is 300k.
  • Users are completely offline for longer times.

We used Flutter to create a React native app, see https://flutter.dev It is a UI toolkit. Why did we use Flutter?

  • It is cross platform mobile.
  • It uses Dart, which is statically typed, making code analysus much easier.
  • Google seems to be prioritizing developer user experience, it really shows of quite a bit.

Dart has asynchronous code as a first class citizen. Quite different from Python. Runtime reflection (pdb) is unavailable. It has good ergonomics, with generics and closures. It is a driving force behind the Flutter toolkit.

Thoughts on mobile development:

  • It is a lot to take in.
  • You need to understand lots of languages, for us: kotlin/java, swift/obj-c, and Dart.
  • No idea how to test platform integration.
  • Native libraries are managed using Cocoapods/Gradle. Flutter drives those. Setting it up is yet another new thing to learn.
  • There are lots of inconveniences, like how do you read sqlite off a device, because that is how we store some of the info?
  • Also inconsistencies: if the app works on an emulator, that does not mean it works on a device.

On the server side:

  • Plone operating as a Webdav server
  • We need to support OIDC (mod_openids/oauth2)
  • Not many writes, maybe 100-1000 per day, but lots of reads.
  • 20k+ devices daily

Alternatives to the server/protocol could be nice:

  • Honestly, are there any standards other than WebDav?
  • An S3 api would be reasonable.
  • So ideas are welcome, let me know.
  • We have been working on prototypes with guillotina_webdav + openidc.

The good parts of Flutter:

  • UI/UX development is very fast, with lots of widgets. We had two developers who were used to Angular, and they took it up quite fast.
  • The bridge to native code (Pigeon) is straight forward.
  • Drift is an amazing sqlite library.
  • Riverpod for state management
  • Dependency management is good (flutter pub). You can tell that they learned a ton from others, like pip. Except that very occasionally the package cache is broken so you need to clean it.
  • They have a good community, with add-ons.

The not so good parts of Flutter:

  • Inconsistent platform features, like WorkManager (Android) versus NSUrlSession (iOS)
  • Dependency churn: often new versions come in, which you then need to check.

The mobile app:

  • We are still wrapping up the remote file operations.
  • After we deploy into production, we will improve the UI.

Yes, we hope to open source the synchronization framework, and maybe the foreground/background transferring subsystem. Yes, we have built a Flutter web-app of this, but it looks just like the mobile app currently. Needs a separate layout really. No, we have not done a Desktop app.

Podcast "The Plone Newsroom" launched

Posted by Starzel.de on September 02, 2021 10:02 AM

Brought to you by Philip Bauer of https://www.starzel.de and Fred van Dijk of https://zestsoftware.nl

Watch it on Youtube Channel https://www.youtube.com/channel/UCPo8CirtIfvt8GO4rm-dD4g

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released!

Posted by kitconcept GmbH on April 12, 2021 09:24 AM

Two weeks ago we cut two important releases on the road to Plone 6. Plone REST API 7 introduces a new link integrity feature for blocks-based pages. Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

stephen leonardi 5CH1TNfcZoo unsplash Photo by Stephen Leonardi on Unsplash

Plone REST API 7

Linking pages is one of the core idea of the world wide web. Keeping links within a website intact is therefore one of the core features that any Content Management System needs to provide.

One of the core features of Plone has always been that editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unigue IDs (UUIDs) instead of relative or absolute paths when adding a link to another page. Plone uses “portal transforms” internally to rewrite those links in RichText fields on save operations.

Plone 6 (aka Volto) introduces blocks-based page layouts, that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout allows more complex page layouts. Though, because of this change, the existing portal transforms mechanism that rewrites links to UUIDs did not work any longer.

volto blocks edit mode Blocks-based Volto page in edit mode (with kitconcept-blocks-grid)

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020, since they needed that for a client project.

I started to draft a possible solution and wrote a first prototype. Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy.

Andrea Cecchi from RedTurtle joined our efforts and refactored the resolveUID algorithm into a blocks transformer, that we started to use for other use cases and that made the resolveUID transformer more generic and flexible.

After that, we cut a first plone.restapi 7 alpha release and entered a longer period of quality assurance and testing.

Two weeks ago, after we tested the new feature in multiple projects at kitconcept, I cut a final release of Plone REST API 7.

After the first alpha release, the ResolveUID transformation was added to links and images. In addition plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

We also added a “smart fields” concept that allows integrators to mark a blocks field as searchableText field.

A new @contextnavigation endpoint was added that allows for local navigations. We enhanced the navigation endpoint to expose an optional navigation title (“nav_title”) field.

Since the final 7 release, we cut six more releases and REST API 7 was included in Plone 5.2.4 release.

Volto 12

gaelle marcel vrkSVpOwchk unsplash Photo by Gaelle Marcel on Unsplash

Tiberiu and Victor worked on a new Volto configuration registry which is the new central point to store and retrieve Volto configurations. The configuration registry is a singleton that ensures a setting only exists once. With more an more Volto add-ons that started to depend on each other, we started to run into circular dependencies issues that we can avoid now with the new configuration registry.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Plone 6

Plone REST API 7 and Volto 12 were two very important releases on our road to Plone 6.

We plan to cut another Plone REST API 8 branch and release for Plone 6, which will support Python 3 only.

Volto will continue to move at very high pace towards Plone 6. We started to organize a series of Plone 6 “Micro-Sprints” to push things further.

Check out the Volto Roadmap on github for more details.

Implementing user epics with BPMN

Posted by Asko Soukka on March 21, 2021 12:00 PM

Many user stories start simple. For example: “A user self registers into a course”. The final version, of course, tend to be more like: “A new user, after being verified to be an authentic real person, creates their new user account, immediately forgets and resets their new password, finally logs into the system, finds the course again, completes payment for the registration, and is then eventually enrolled into the course.” This is no longer a user story, but a series of stories supposed to be completed in a specific sequence. Let’s call these user epics.

An epic is a sequence a user stories

Individual user stories, by their definition, should be relatively straightforward to implement. For example, a stereotypical user story may require interaction with specific user interface element to start the story, following one or more views to gather the required user input to continue the story, finally ending with user’s goal for the story. Where did the user come from, or where does the user continue afterwards, is simply not part of the story. On purpose.

But user epics are different.

Epics must know the consequences of users' actions

An implementation of a user epic must know where the user came from, and must be able to decide, which stories the user should be guided to complete next. Epic should also cover the consequences of user failing to complete a story, or completely abandoning their fate. Therefore implementing an epic is not only about managing state, but also about programming the often complex business logic. Both being among the least fun parts of software development.

As an insult to injury, less and less are users doing their work alone, more and more they are being accompanied by various forms of automation. So, not only should an implemented epic be aware of the actions of its user, but it should also be able to react on known results or side effects of automation – anything from synchronous function calls to asynchronously executed software robots.

With just code, implementations for this kind of epics could get dirty and hard to understand, fast.

Epics must know the consequences of users' actions

Offloading state and logic to BPMN engine

It’s not a coincidence that my example user epics above are drawn as BPMN diagrams. That’s what I have been doing for the last couple of months in our latest project. We had to modernize a process, where users were found to require a lot of hand-holding to be able complete all the required steps and in the right order.

Also, with the previous implementation, when users got lost in the middle of their epics, or an unexpected system error blocked them, it was often surprisingly hard to figure out what really had happened.

This time all this should be different.

The active user task may be anywhere in the current process tree

Whenever a user starts a new epic, a new BPMN modeled process is being started. The model may be as long and complex as needed. It may even be a composition of multiple nested models, resulting in an arbitrarily deep process tree when executed. But this is not a problem, because this is exactly what dedicated BPMN process execution engines have been built for.

The fun part: With a BPMN engine executing our user epics directly from their BPMN models, we don’t really need to “implement” them at all. Instead, we:

  1. Implement user interface routes and views for individual user tasks.
  2. Make the views to read their preconditions from the process engine state.
  3. Eventually complete the task at the engine (with user input when required).
  4. Poll the engine for the next available task and redirect the user there.

Now our user task views are standalone, with no magical dependencies between them to code and maintain.

Plone, Volto and Camunda FTW

I would not be writing this, unless all the above could not be implemented by using open source software only:

  • Because every application seems to need a CMS (sooner or later), we build on top of Plone, the open source CMS we trust. Thanks to its powerful and complete REST API, we can do all the integrations we need with just HTTP requests – no need to know about Plone internals.

  • For modern user experience, we use Volto – ReactJS based web experience on top of Plone REST API (with Razzle based server side rendering support). Every part of Volto is easily customizable and extensible. For example, custom server side routes, like the ones we need to authorize and proxy calls to BPMN engine, are common and well documented use case.

  • Finally, for the BPMN engine, we choose Camunda BPM Community Edition. While the open source version of Camunda is probably missing countless convenient features of their enterprise offering, its BPMN engine and REST API come uncompromised.

Thanks to all above, it’s not a (technical) problem at all that users tend to forget their passwords just after sign up. Now the process engine remembers were they came from, and what they were supposed to do, just after resetting their password one more time…

With a reusable sub process, password change can part of any epic

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\[email protected]]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'[email protected]': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'[email protected]': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone [email protected]'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor , 2) Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.

  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.

  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.

  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.

All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.

If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.

Plone Beethoven Sprint 2019

Posted by kitconcept GmbH on September 25, 2019 09:00 AM

beethoven sprint group image

21 developers from nine different countries gathered in Bonn, Germany between June 20th and 24th to work on implementing the upcoming Plone 6. The sprint at the office of the kitconcept GmbH has been declared a “strategic sprint” by the Plone framework team. Sprint topics included working on Volto, the Plone REST API, and Guillotina.

Volto

Volto is a new ReactJS-based frontend for Plone. It was started by Rob Gietema in 2017 and it is actively developed since then. Volto will become the default frontend for Plone 6. It implements a complete new user interface called Pastanaga UI, which was developed by Albert Casado.

Victor Fernandez de Alba and Rob Gietema led the efforts to further enhance Volto together with Paul Roeland, Nicola Zambello, Piero Nicolli, Janina Hard, Jakob Kahl, Thomas Kindermann, Maurizio Delmonte, Stefania Trabucchi, Rodrigo Ferreira de Souza, Fred van Dijk, and Steffen Lindner.

Pastanaga UI Toolbar

The new editing toolbar has been part of Volto right from the start. Though, so far we did not fully implement every detail that Albert has imagined for the Pastanaga UI toolbar. Victor has been working on this for quite a while already. During the sprint, he integrated his work and polished it. Victor and Rob also worked on a new sidebar that shows up on the right side of the Volto user interface that holds additional information like page metadata and controls the settings of the individual tiles.

volto toolbar The new Volto edit toolbar

New Tiles: Collection, Slider and Table

At the Plone conference in Tokyo, we came up with a list of tiles for the new Volto composite page editor that we would like to implement. On top of that list were the listing/collection tile, the slider tile and the table tile. Piero, Rodrigo, and Rob started to work on the listing/collection tile that is supposed to become a replacement for the Collection Type in Plone.

Jakob and Janina worked on copying over the code of a slider tile that was developed by kitconcept. Rob implemented the table tile. Before you could ask Rob about the progress, he already finished his work, like always…

volto table The new Volto table tile

Accessibility

Accessibility has always been a first class citizen in the Plone community. Plone is used by many government and public websites where accessibility is a basic requirement. Paul, Thomas, Timo, and Victor worked on improving the accessibility of Volto. Our goal is to fully support the WCAG 2.1 standard with the “AA” level of conformance.

Timo set up ESlint-based static code analysis checks and Cypress-based axe accessibility tests. Paul, Thomas and Victor then started to fix the reported accessibility issues. At the end of the sprint, we were able to fix all the reported violations. Our CI build now fails if new accessibility violations are committed.

Documentation

Sven Strack, the Plone documentation team leader, joined our sprint remotely from Amsterdam to discuss how to maintain and structure the Volto documentation in the future. We all agreed that we will use Markdown as format for our docs, since this is the de-facto standard in the JavaScript community as well as in most other Open Source communities. As much as we like Restructured Text, it seems that Markdown won that battle.

We currently use MKdocs for the Volto documentation and Styleguidist to document our React components. There are lots of different tools to choose from, Docz, MKDocs, MDX, Gatsby to just name a few. We agreed that we need to do more research to make an educated decision regarding our doc toolchain.

There will be a Google Season of Docs project this year where a technical writer will help us to enhance the Volto docs further. More Volto Features Nicola added a feature to display the currently used Volto version in the Plone control panel, he added animations for page transitions and added an Italian translation for Volto. Together with Victor, he worked on a toast component for user notifications.

More Volto Features

Piero worked on improving and fixing the event type view and added styling for the listing tile.

Victor and Steffen worked on the image sidebar.

Rodrigo added a feature that allows the user to use SHIFT+RETURN to create a new line in a Volto text. By default Volto creates a new text tile on RETURN. Rodrigo also looked into the Add-ons control panel work that Eric Steele started some time ago.

Nilesh joined us remotely from India and continued his work on the users and groups control panel for Volto.

Steffen fixed a bug in the subjects search. Rodrigo and Thomas looked into our Cypress-based acceptance tests.

Fred looked into what consequences it would have for us to get rid of the description tile and how we could customize the domain in Plone to allow to send portal emails via the Volto frontend.

Stefania and Maurizio worked on a product definition of Volto and an elevator pitch for clients and users. Together with Paul, Fred and Timo, they had a longer discussion about how to position Volto and Plone 6 in the market.

Plone REST API

The Plone REST API builds the foundation for the new Volto frontend and Plone 6. It has been under active development in the last four years and seen more than 80 releases in that time period. The REST API is stable, battle-tested and production-ready and part of the Plone core since the release of Plone 5.2.

Thomas Buchberger and Lukas Graf led the REST API efforts during the sprint and worked with Carsten Senger, Roel Bruggink, Janina Hard and Timo Stollenwerk to further enhance the REST API, fix bugs, improve the documentation, and the error handling.

API Team in the graden Plone REST API discussion in the garden

Querystring Endpoint for the Collection Type

One of the most important new features, that has been developed during the sprint, is the new querystring endpoint. This endpoint will allow us to implement the Collection type for the Volto frontend. Lukas implemented this essential feature with help from Rob, who took care of the frontend implementation in Volto.

New Features, Bugfixes, and first time contributions

Thomas added support for the retrieval of additional metadata fields in summaries in the same way as in search results. Lukas made the @types endpoint expandable and the @users endpoint easier to customize. Janina fixed setting the effective date on workflow transitions. This was her first contribution to the Plone core. On the same day she signed the contributor agreement, got her fix merged and released, which is quite an accomplishment for a single day. Way to go Janina!

In addition to that, the team fixed lots of smaller bugs (#780, #729, #764, #755, #777, #732, #738) during the sprint. REST API Error handling Carsten, Nathan, Thomas, Lukas and Timo had a longer discussion about error handling in the Plone REST API and about how we can improve things. We considered and discussed implementing RFC7807 and decided against adopting it.

Developer in office Sprint discussions…

The RFC did not see much adoption and it only slightly differs from what we are already doing. A breaking change does not seem to be worth the effort. We decided to create a separate error reporting component unifying the error reporting and ensuring consistency. We also started to fix a few inconsistencies right at the sprint.

API Documentation

Guillotina is using Swagger to document its API. Nathan explained the approach he took for that and Carsten started to integrate Swagger to generate dynamic api docs in plone.restapi.

REST API Releases

Most of the work we did on Plone REST API during the sprint has been released in Plone REST API 4.1.4, 4.2.0, 4.3.0, and 4.4.0.

Dexterity Site Root

Roel Bruggink joined us on Saturday and Sunday to continue his work on turning the Plone site root into a Dexterity content object. He fixed some complex recursion and acquisition problem when migrating the site root to Dexterity. His Plone Improvement Proposal (PLIP #2454) has been accepted by the Plone framework team and it will build the foundation for a new Plone 6 branch that we plan to cut this year.

Guillotina

Guillotina is a new AsyncIO REST data API. It has been written from the scratch by Ramon Navarro Bosch and Nathan van Gheem. It can easily scale billions of objects via its REST API. With the CMS addon, it is compatible with the Plone REST API. Therefore you can run Volto on a Guillotina backend and Guillotina might be able to replace our existing Plone backend in the future.

Guillotina Developer The Guillotina master thinking hard. :)

Nathan and Ramon worked on Guillotina 5. The Guillotina 5 release brought Python 3.7 support and the use of the context vars APIs. They refactored quite a bit, moved parts of addons into core(swagger, caching, pg catalog), implemented OOTB PostgreSQL catalog, added a Helm/Kubernetes/Docker configurations for the Guillotina CMS addon that can be configured to act as a Volto backend, worked on improving caching and filestorage.

Websockets for Plone

At the Plone Open Garden 2019, Asko Soukka started to work on bringing websocket support to Plone. Websockets allow bi-directional communication between the server and the browser. With websockets, the server can send messages and notifications to the browser. This can be used to implement notification systems or collaborative editing to just name a few possible applications.

Asko continued his work during the Beethoven sprint and created an example where comments that are added to Plone automatically pop up as toast notifications for other users. He published a Docker image with a Twisted-based ZServer and implemented the foundation to use that feature for gatsby-source-plone. This will allow us to create an extension that will allow editors to live-preview changes they do in Plone in a GatsbyJS site.

Developer in Office

Karaoke, Barbecue and Garden Parties

A Plone sprint is much more than just programming. Collaborating, meeting old friends, talk, discuss and having a good time together are the ingredients that make a Plone sprint. We started our sprint on Thursday with a free React and Volto training for newbies. At the evening when everybody arrived, we went to a local Karaoke bar for a revival of the epic Karaoke sessions we had in Tokyo last year.

Karaoke Karaoke at the “Nyx”

On Friday evening we ordered Pizza and hacked the night away at the office and our garden. On Saturday we went to the “Arithmeum”, a mathematics museum owned by the Research Institute for Discrete Mathematics at the University of Bonn.

Arithmeum Our tour at the “Arithmeum”

Afterwards we had a garden party with a barbecue and we spend the afternoon sitting in the garden and the office and continued to drink and hack on Plone.

Developer in garden Barbecue in the garden of the kitconcept office

Summary

The Beethoven Sprint 2019 was a major success. We were able to start working on the main missing pieces for Volto. The new edit toolbar is in place and the foundation for the new tiles/metadata toolbar on the right side of the UI has been finished. We also implemented a new table tile.

We implemented a proof-of-concept for the new listing / collection tile and implemented the required querystring endpoint in Plone REST API. Both Volto and the Plone REST API got lots of enhancements and bugfixes.

Both the Plone REST API and Volto are stable and used in production. We will use the next month to further enhance Volto and implement the missing bits and pieces. Our plan is to present Volto 4 at the Plone conference in Ferrara at the end of this year and cut a Plone 6 branch afterwards that contains the Dexterity Site root, folderish content types and other enhancements that are needed make Volto the default frontend for Plone 6.

Guillotina and the Websockets support for ZServer together with Gatsby as another frontend for Plone are amazing projects by the smartest folks in our community. Our diversity is one of our main strength and we are looking forward what the future will bring!

We had a great time at the sprint. Thank you to everyone who attended! We thank the Plone and Python community for their ongoing support. The Plone Foundation, the Python Software Foundation, and the German Python Software Verband made this event possible with their sponsorship as well as our company sponsors Abstract, Starzel, Werkbank and Zest.

We are looking forward to see you all in Bonn next year!

Developer in garden The “Hofgarten” near the kitconcept office

Digital Signage management with Plone and GatsbyJS

Posted by Asko Soukka on September 01, 2019 09:00 AM

Sometimes complex problems have simple solutions, and this was one of those times. At work, we are piloting a new digital signage solution for our university. Unfortunately, the new system lacks in permission management, and for a while, it looked like we were unable to safely delegate digital signage content management to departments.

Lucky us that we knew a system that is very good in permission management: Plone.

Plone – An enterprise open source content management system

Plone is an open source content management system based on hierarchical object database. That’s why Plone excels in managing many kind of content and related permissions in hierarchies. That’s also why Plone has been a good fit for such hierarchical organizations where content management follows the hierarchy of the organization itself.

Plone excels in managing content and related permissions in hierarchies

So, how we are going to solve the shortcomings of our digital signage system with Plone and GatsbyJS? In brief, we

  • let our users to manage their digital signage content on Plone in the simples possible way: every display group has their own folder on Plone and the responsible users have required rights to manage content in their folders

  • we use GatsbyJS project to fetch that content from Plone and build simple digital signage content players hosted as static web pages

  • we use our on-premises GitLab CI and GitLab pages to build and host those web pages; a Plone feature called content rules is used to trigger a new updating build after each content update.

With this setup, instead of letting the content managers to directly manage the digital signage system (where we were unable to restrict their permissions to their own display groups), we configure the system to display those GatsbyJS built sites and let the users manage their digital signage content on Plone – on the content management system they are already familiar with.

Simple digital signage content loop can be populated with just dropping images into a Plone folder

And that’s just the beginning. Thanks to Plone’s customizable structured content types, we already added new content field for setting the display time of each content page. And in the future we can use this to add more configuration options to ensure that the content is displayed as intended by the content managers.

Plone supports customizing structured content fields directly form the browser

Also built-in content types like first class Image type can be enhanced with custom fields

gatsby-theme-ds-player

Show the code or it didn’t happen!

To be honest, our current solution is so simple that there is not that much to show.

Anyway, I put together a simple open source version where similar GatbyJS built web site is created from given Plone folder so that each subfolder has its own digital signage player to loop through the images dropped into that folder.

I packaged all this into a GatsbyJS “theme” plugin gatsby-theme-ds-player preconfigured to be best compatible with a Plone source using gatsby-source-plone. There is also a live demo of the result, automatically built from the theme package repository at Travis-CI:

I got interviewed about Plone 5.2

Posted by Starzel.de on July 19, 2019 02:15 PM

Click here to read the interview on plone.org.

Using Glitch for Pyramid Learning

Posted by Jazkarta Blog on May 09, 2019 08:02 PM

Screenshot: Pyramid Quick Tutorials showing pyramid-view-decorators, pyramid-view-predicates, pyramid-view-renderers and pyramid-asset-specifications

At this year’s Jazkarta sprint we decided to invest some time in creating Pyramid learning resources. We are fans of the Pyramid web framework, and we felt that it would be nice to work on something that would make it easier to get started with, especially for novice developers.

Enter Glitch. This is a platform that is designed for letting users explore web apps created by other Glitch users. When you see an application that does something that you want to do, you can “remix” it, which means that an exact copy, with all its code and resources, is created in your own workspace (which is some sort of Docker container). You can then change it in any way you like. This is very powerful, because you get a working application that is hosted for you, and every code change that you make is immediately reflected in the web app page.

This turns out to be really useful for teaching web application development concepts, because students can see the code for the example, read an explanation, and then right away remix the code to try out their changes, which they can see working instantly. No need for setting up a workspace, checking out code, or running a web server. Just try out an idea and see the result right there, or share it with other people. Like “view HTML source” but with code, which is one of Glitch’s goals.

We decided to create a Glitch collection that would showcase a few of Pyramid’s most distinctive features, along with a simple Hello World style tutorial. The first thing was to verify that Pyramid could be installed. Glitch’s main development target is Node.js, but its containers install Python and they allow the user to run pip, so it’s possible to install most PyPI packages including Pyramid. (For those who might be wondering: sorry, Plone is not installable, because it needs system dependencies that are not installed by Glitch.)

Glitch supports a setup file named glitch.json, which is what allowed us to install Pyramid:

{
 "install": "pip3 install --user pyramid",
 "start": "python3 pyramid_app.py"
}

In the json above, you can see that we install Pyramid using pip, and we can add any other dependencies for our project there. We can then use the “start” key to tell Glitch how to start our application.

Once we did that, the only other thing needed was to create our Pyramid app, which in this case uses code directly lifted from the trypyramid.com site:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

def hello_world(request):
    return Response('Hello Glitch')

if __name__ == '__main__':
    with Configurator() as config:
       config.add_route('hello', '/')
       config.add_view(hello_world, route_name='hello')
       app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 3000, app)
    server.serve_forever()

That’s it. Glitch automatically runs the application (the only caveat is that it needs to use port 3000). You can configure it so that any code change automatically restarts the app. After this, if a potential new user sees your project, she can just use the remix button and have a copy made in her workspace. Any change that she makes, like changing the hello text, will be immediately visible on the page.

Overall, we found Glitch to be a very intuitive and easy to use service. You can see the collection of simple tutorials that we created here:

Pyramid Quick Tutorials

 

Jazkarta Goes To LA

Posted by Jazkarta Blog on May 09, 2019 07:47 PM

The Jazkarta team at LAX-C, a city block sized Asian food distributor and restaurant supply

This year Jazkarta’s annual sprint was held in the city of Los Angeles, graciously hosted by Alec Mitchell who lives a block away from LA’s City Hall.

View of the Los Angeles City Hall from Los Angeles Street

We had fun exploring the city – Union Station, Chinatown, LAX-C (pictured above, a gigantic Asian food distributor and restaurant supply), Grand Central Market, Griffith Observatory, and the Hollywood Farmers’ Market where we bought lots of delicious fruit.

Citrus and strawberries from the Hollywood Farmers' Market

We attended an improv comedy performance at the Upright Citizens Brigade Theater, ate lots of tacos, had original Phillipe’s French Dip sandwiches, visited brew pubs and drank lots of local beer.

The Jazkarta team in front of the Highland Park Brewery.

We even had a private tasting of cheese, beer, and wine pairings.

Plate with five different cheeses and a variety of fruits and nuts

Oh, and we also sprinted on a variety of topics!

Alec and Jesse Snyder ported several of our Plone add-ons to Python 3 and Plone 5.2:

Carlos de la Guardia created a set of Pyramid Quick Tutorials on Glitch – more about that coming in another post.

Matthew Wilkes and Jesse researched the best way to set up Pyramid functional tests using pytest and webtest in an efficient manner. They were unhappy with their previous model, which relied on in-memory databases. They settled on a pattern that switched out private variables of the sessionmaker to join a longer-running transaction for each test. This is more self-contained than adapting the session setup machinery in the app itself, and works transparently on different database backends. Matthew made a gist of the relevant code.

Alec made a prototype of an app to provide faceted search of bibliography references kept in Zotero. The app has a Elasticsearch backend and a React front end. It uses the Zotero API to pull the references into Elasticsearch for querying.

Carlos investigated what it would take to move the CastleCMS quality checking features into a Plone add-on. It’s a work in progress.

Left to right: Carlos, Sally, Matthew, Alec, Jesse and Nate in front of a painting of the Griffith Observatory

It’s wonderful to be able to work together in person, instead of remotely from our home offices spread across 3 countries and 2 continents. We try to do it once a year and always have a fantastic time. This year was no exception.

© Plone Community and friends.

Blog entries aggregated on this page are owned by, and represent the opinion of the author.