Skip to content
This repository has been archived by the owner on Aug 4, 2023. It is now read-only.

Simplify loader #167

Closed
wants to merge 5 commits into from
Closed

Simplify loader #167

wants to merge 5 commits into from

Conversation

obulat
Copy link
Contributor

@obulat obulat commented Sep 8, 2021

Fixes

Fixes WordPress/openverse#1712 by @obulat

Description

This PR creates a DbColumn class that defines a database column, and describes its behavior on conflict, when updating data. A list of columns for different media types and loading and main database (openledger, also referred to as upstream) makes it easy to create SQL query strings.
The PR moves the mediatype-specific columns to the end of the loading table, so that the column lists for different media types can be created simply by concatenating a list of common columns with a list of media type - specific columns. This caused a lot of tests to break.
This PR also removes legacy workflows for cleaning up DB tables, used before the ImageStore was used to clean up media data. They are no longer necessary since all the back up data we have has already been cleaned, and the data we collect from the APIs will be cleaned up through ImageStore.

Technical details

Tests

Screenshots

Checklist

  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@obulat obulat requested a review from a team as a code owner September 8, 2021 19:20
@dhruvkb dhruvkb added this to Needs review in Openverse Sep 8, 2021
Copy link
Member

@dhruvkb dhruvkb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, keeping media-specific columns at the end is a great idea. From an overview LGTM. I'll review more comprehensively later today.

@obulat
Copy link
Contributor Author

obulat commented Sep 10, 2021

Closing this in favor of a better-organized #168

@obulat obulat closed this Sep 10, 2021
Openverse automation moved this from Needs review to Done! Sep 10, 2021
@obulat obulat deleted the simplify_loader branch September 17, 2021 05:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
No open projects
Openverse
  
Done!
Development

Successfully merging this pull request may close these issues.

[Infrastructure] Simplify the TSV to Postgres loading process
2 participants