WIP - Pydantic v2 support (#1238)

* WIP

* WIP - make test_model_definition tests pass

* WIP - make test_model_methods pass

* WIP - make whole test suit at least run - failing 49/443 tests

* WIP fix part of the getting pydantic tests as types of fields are now kept in core schema and not on fieldsinfo

* WIP fix validation in update by creating individual fields validators, failing 36/443

* WIP fix __pydantic_extra__ in intializing model, fix test related to pydantic config checks, failing 32/442

* WIP - fix enum schema in model_json_schema, failing 31/442

* WIP - fix copying through model, fix setting pydantic fields on through, fix default config and inheriting from it, failing 26/442

* WIP fix tests checking pydantic schema, fix excluding parent fields, failing 21/442

* WIP some missed files

* WIP - fix validators inheritance and fix validators in generated pydantic, failing 17/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - working on proper populating __dict__ for relations for new schema dumping, some work on openapi docs, failing 13/442

* WIP - remove property fields as pydantic has now computed_field on its own, failing 9/442

* WIP - fixes in docs, failing 8/442

* WIP - fix tests for largebinary schema, wrapped bytes fields fail in pydantic, will be fixed in pydantic-core, remaining is circural schema for related models, failing 6/442

* WIP - fix to pk only models in schemas

* Getting test suites to pass (#1249)

* wip, fixing tests

* iteration, fixing some more tests

* iteration, fixing some more tests

* adhere to comments

* adhere to comments

* remove unnecessary dict call, re-add getattribute for testing

* todo for reverse relationship

* adhere to comments, remove prints

* solve circular refs

* all tests pass 🎉

* remove 3.7 from tests

* add lint and type check jobs

* reforat with ruff, fix jobs

* rename jobs

* fix imports

* fix evaluate in py3.8

* partially fix coverage

* fix coverage, add more tests

* fix test ids

* fix test ids

* fix lint, fix docs, make docs fully working scripts, add test docs job

* fix pyproject

* pin py ver in test docs

* change dir in test docs

* fix pydantic warning hack

* rm poetry call in test_docs

* switch to pathlib in test docs

* remove coverage req test docs

* fix type check tests, fix part of types

* fix/skip next part of types

* fix next part of types

* fix next part of types

* fix coverage

* fix coverage

* fix type (bit dirty 🤷)

* fix some code smells

* change pre-commit

* tweak workflows

* remove no root from tests

* switch to full python path by passing sys.executable

* some small refactor in new base model, one sample test, change makefile

* small refactors to reduce complexity of methods

* temp add tests for prs against pydantic_v2

* remove all references to __fields__

* remove all references to construct, deprecate the method and update model_construct to be in line with pydantic

* deprecate dict and add model_dump, todo switch to model_dict in calls

* fix tests

* change to union

* change to union

* change to model_dump and model_dump_json from dict and json deprecated methods, deprecate them in ormar too

* finish switching dict() -> model_dump()

* finish switching json() -> model_dump_json()

* remove fully pydantic_only

* switch to extra for payment card, change missed json calls

* fix coverage - no more warnings internal

* fix coverage - no more warnings internal - part 2

* split model_construct into own and pydantic parts

* split determine pydantic field type

* change to new field validators

* fix benchmarks, add codspeed instead of pytest-benchmark, add action and gh workflow

* restore pytest-benchmark

* remove codspeed

* pin pydantic version, restore codspeed

* change on push to pydantic_v2 to trigger first one

* Use lifespan function instead of event (#1259)

* check return types

* fix imports order, set warnings=False on json that passes the dict, fix unnecessary loop in one of the test

* remove references to model's meta as it's now ormar config, rename related methods too

* filter out pydantic serializer warnings

* remove choices leftovers

* remove leftovers after property_fields, keep only enough to exclude them in initialization

* add migration guide

* fix meta references

* downgrade databases for now

* Change line numbers in documentation (#1265)

* proofread and fix the docs, part 1

* proofread and fix the docs for models

* proofread and fix the docs for fields

* proofread and fix the docs for relations

* proofread and fix rest of the docs, add release notes for 0.20

* create tables in new docs src

* cleanup old deps, uncomment docs publish on tag

* fix import reorder

---------

Co-authored-by: TouwaStar <30479449+TouwaStar@users.noreply.github.com>
Co-authored-by: Goran Mekić <meka@tilda.center>
This commit is contained in:
collerek
2024-03-23 19:28:28 +01:00
committed by GitHub
parent 3a206dd8dc
commit 500625f0ec
294 changed files with 8132 additions and 9311 deletions

View File

@ -1,31 +1,20 @@
# type: ignore
from typing import List, Optional
import databases
import pytest
import sqlalchemy as sa
from pydantic.typing import ForwardRef
from sqlalchemy import create_engine
from typing import ForwardRef, List, Optional
import ormar
from ormar import ModelMeta
from tests.settings import DATABASE_URL
import pytest
from tests.lifespan import init_tests
from tests.settings import create_config
base_ormar_config = create_config()
metadata = sa.MetaData()
db = databases.Database(DATABASE_URL)
engine = create_engine(DATABASE_URL)
TeacherRef = ForwardRef("Teacher")
class BaseMeta(ormar.ModelMeta):
metadata = metadata
database = db
class Student(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -35,14 +24,11 @@ class Student(ormar.Model):
class StudentTeacher(ormar.Model):
class Meta(BaseMeta):
tablename = "students_x_teachers"
ormar_config = base_ormar_config.copy(tablename="students_x_teachers")
class Teacher(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -58,8 +44,7 @@ CountryRef = ForwardRef("Country")
class Country(ormar.Model):
class Meta(BaseMeta):
tablename = "countries"
ormar_config = base_ormar_config.copy(tablename="countries")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=128)
@ -70,8 +55,7 @@ class Country(ormar.Model):
class City(ormar.Model):
class Meta(BaseMeta):
tablename = "cities"
ormar_config = base_ormar_config.copy(tablename="cities")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=128)
@ -83,17 +67,13 @@ class City(ormar.Model):
Country.update_forward_refs()
@pytest.fixture(autouse=True, scope="module")
def create_test_database():
metadata.create_all(engine)
yield
metadata.drop_all(engine)
create_test_database = init_tests(base_ormar_config)
@pytest.mark.asyncio
async def test_double_relations():
async with db:
async with db.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
t1 = await Teacher.objects.create(name="Mr. Jones")
t2 = await Teacher.objects.create(name="Ms. Smith")
t3 = await Teacher.objects.create(name="Mr. Quibble")
@ -150,8 +130,8 @@ async def test_double_relations():
@pytest.mark.asyncio
async def test_auto_through_model():
async with db:
async with db.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
england = await Country(name="England").save()
france = await Country(name="France").save()
london = await City(name="London", country=england).save()

View File

@ -1,29 +1,23 @@
# type: ignore
from typing import List
from typing import ForwardRef, List, Optional
import databases
import ormar
import pytest
import pytest_asyncio
import sqlalchemy as sa
from pydantic.typing import ForwardRef
from sqlalchemy import create_engine
import ormar
from ormar import ModelMeta
from ormar.exceptions import ModelError
from tests.settings import DATABASE_URL
metadata = sa.MetaData()
db = databases.Database(DATABASE_URL)
engine = create_engine(DATABASE_URL)
from tests.lifespan import init_tests
from tests.settings import create_config
base_ormar_config = create_config()
PersonRef = ForwardRef("Person")
class Person(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -38,9 +32,7 @@ ChildFriendRef = ForwardRef("ChildFriend")
class Child(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -54,15 +46,11 @@ class Child(ormar.Model):
class ChildFriend(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
class Game(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -71,17 +59,13 @@ class Game(ormar.Model):
Child.update_forward_refs()
@pytest.fixture(autouse=True, scope="module")
def create_test_database():
metadata.create_all(engine)
yield
metadata.drop_all(engine)
create_test_database = init_tests(base_ormar_config)
@pytest_asyncio.fixture(scope="function")
async def cleanup():
yield
async with db:
async with base_ormar_config.database:
await ChildFriend.objects.delete(each=True)
await Child.objects.delete(each=True)
await Game.objects.delete(each=True)
@ -93,9 +77,7 @@ async def test_not_updated_model_raises_errors():
Person2Ref = ForwardRef("Person2")
class Person2(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -116,14 +98,10 @@ async def test_not_updated_model_m2m_raises_errors():
Person3Ref = ForwardRef("Person3")
class PersonFriend(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
class Person3(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -146,17 +124,13 @@ async def test_not_updated_model_m2m_through_raises_errors():
PersonPetRef = ForwardRef("PersonPet")
class Pet(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Person4(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -165,9 +139,7 @@ async def test_not_updated_model_m2m_through_raises_errors():
)
class PersonPet(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
with pytest.raises(ModelError):
await Person4.objects.create(name="Test")
@ -180,25 +152,25 @@ async def test_not_updated_model_m2m_through_raises_errors():
def test_proper_field_init():
assert "supervisor" in Person.Meta.model_fields
assert Person.Meta.model_fields["supervisor"].to == Person
assert "supervisor" in Person.ormar_config.model_fields
assert Person.ormar_config.model_fields["supervisor"].to == Person
assert "supervisor" in Person.__fields__
assert Person.__fields__["supervisor"].type_ == Person
assert "supervisor" in Person.model_fields
assert Person.model_fields["supervisor"].annotation == Optional[Person]
assert "supervisor" in Person.Meta.table.columns
assert "supervisor" in Person.ormar_config.table.columns
assert isinstance(
Person.Meta.table.columns["supervisor"].type, sa.sql.sqltypes.Integer
Person.ormar_config.table.columns["supervisor"].type, sa.sql.sqltypes.Integer
)
assert len(Person.Meta.table.columns["supervisor"].foreign_keys) > 0
assert len(Person.ormar_config.table.columns["supervisor"].foreign_keys) > 0
assert "person_supervisor" in Person.Meta.alias_manager._aliases_new
assert "person_supervisor" in Person.ormar_config.alias_manager._aliases_new
@pytest.mark.asyncio
async def test_self_relation():
async with db:
async with db.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
sam = await Person.objects.create(name="Sam")
joe = await Person(name="Joe", supervisor=sam).save()
assert joe.supervisor.name == "Sam"
@ -215,8 +187,8 @@ async def test_self_relation():
@pytest.mark.asyncio
async def test_other_forwardref_relation(cleanup):
async with db:
async with db.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
checkers = await Game.objects.create(name="checkers")
uno = await Game(name="Uno").save()
@ -242,8 +214,8 @@ async def test_other_forwardref_relation(cleanup):
@pytest.mark.asyncio
async def test_m2m_self_forwardref_relation(cleanup):
async with db:
async with db.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
checkers = await Game.objects.create(name="Checkers")
uno = await Game(name="Uno").save()
jenga = await Game(name="Jenga").save()

View File

@ -1,42 +1,30 @@
import asyncio
from typing import Optional
import databases
import pytest
import sqlalchemy
import ormar
from tests.settings import DATABASE_URL
import pytest
database = databases.Database(DATABASE_URL, force_rollback=True)
metadata = sqlalchemy.MetaData()
from tests.lifespan import init_tests
from tests.settings import create_config
base_ormar_config = create_config(force_rollback=True)
class Department(ormar.Model):
class Meta:
tablename = "departments"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="departments")
id: int = ormar.Integer(primary_key=True, autoincrement=False)
name: str = ormar.String(max_length=100)
class SchoolClass(ormar.Model):
class Meta:
tablename = "schoolclasses"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="schoolclasses")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -44,10 +32,7 @@ class Category(ormar.Model):
class Student(ormar.Model):
class Meta:
tablename = "students"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="students")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -56,10 +41,7 @@ class Student(ormar.Model):
class Teacher(ormar.Model):
class Meta:
tablename = "teachers"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="teachers")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -67,13 +49,7 @@ class Teacher(ormar.Model):
category: Optional[Category] = ormar.ForeignKey(Category, nullable=True)
@pytest.fixture(autouse=True, scope="module")
def create_test_database():
engine = sqlalchemy.create_engine(DATABASE_URL)
metadata.drop_all(engine)
metadata.create_all(engine)
yield
metadata.drop_all(engine)
create_test_database = init_tests(base_ormar_config)
async def create_data():
@ -91,35 +67,35 @@ async def create_data():
@pytest.mark.asyncio
async def test_model_multiple_instances_of_same_table_in_schema():
async with database:
async with base_ormar_config.database:
await create_data()
classes = await SchoolClass.objects.select_related(
["teachers__category__department", "students__category__department"]
).all()
assert classes[0].name == "Math"
assert classes[0].students[0].name == "Jane"
assert len(classes[0].dict().get("students")) == 2
assert len(classes[0].model_dump().get("students")) == 2
assert classes[0].teachers[0].category.department.name == "Law Department"
assert classes[0].students[0].category.department.name == "Math Department"
@pytest.mark.asyncio
async def test_load_all_multiple_instances_of_same_table_in_schema():
async with database:
async with base_ormar_config.database:
await create_data()
math_class = await SchoolClass.objects.get(name="Math")
assert math_class.name == "Math"
await math_class.load_all(follow=True)
assert math_class.students[0].name == "Jane"
assert len(math_class.dict().get("students")) == 2
assert len(math_class.model_dump().get("students")) == 2
assert math_class.teachers[0].category.department.name == "Law Department"
assert math_class.students[0].category.department.name == "Math Department"
@pytest.mark.asyncio
async def test_filter_groups_with_instances_of_same_table_in_schema():
async with database:
async with base_ormar_config.database:
await create_data()
math_class = (
await SchoolClass.objects.select_related(
@ -136,7 +112,7 @@ async def test_filter_groups_with_instances_of_same_table_in_schema():
)
assert math_class.name == "Math"
assert math_class.students[0].name == "Jane"
assert len(math_class.dict().get("students")) == 2
assert len(math_class.model_dump().get("students")) == 2
assert math_class.teachers[0].category.department.name == "Law Department"
assert math_class.students[0].category.department.name == "Math Department"

View File

@ -1,32 +1,23 @@
import asyncio
from typing import Optional
import databases
import pytest
import sqlalchemy
import ormar
from tests.settings import DATABASE_URL
import pytest
database = databases.Database(DATABASE_URL, force_rollback=True)
metadata = sqlalchemy.MetaData()
from tests.lifespan import init_tests
from tests.settings import create_config
base_ormar_config = create_config()
class Department(ormar.Model):
class Meta:
tablename = "departments"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="departments")
id: int = ormar.Integer(primary_key=True, autoincrement=False)
name: str = ormar.String(max_length=100)
class SchoolClass(ormar.Model):
class Meta:
tablename = "schoolclasses"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="schoolclasses")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -34,20 +25,14 @@ class SchoolClass(ormar.Model):
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Student(ormar.Model):
class Meta:
tablename = "students"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="students")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -56,10 +41,7 @@ class Student(ormar.Model):
class Teacher(ormar.Model):
class Meta:
tablename = "teachers"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="teachers")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -67,13 +49,7 @@ class Teacher(ormar.Model):
category: Optional[Category] = ormar.ForeignKey(Category, nullable=True)
@pytest.fixture(autouse=True, scope="module")
def create_test_database():
engine = sqlalchemy.create_engine(DATABASE_URL)
metadata.drop_all(engine)
metadata.create_all(engine)
yield
metadata.drop_all(engine)
create_test_database = init_tests(base_ormar_config)
async def create_data():
@ -91,8 +67,8 @@ async def create_data():
@pytest.mark.asyncio
async def test_model_multiple_instances_of_same_table_in_schema():
async with database:
async with database.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
await create_data()
classes = await SchoolClass.objects.select_related(
["teachers__category", "students__schoolclass"]
@ -100,9 +76,10 @@ async def test_model_multiple_instances_of_same_table_in_schema():
assert classes[0].name == "Math"
assert classes[0].students[0].name == "Jane"
assert len(classes[0].dict().get("students")) == 2
assert len(classes[0].model_dump().get("students")) == 2
# since it's going from schoolclass => teacher => schoolclass (same class) department is already populated
# since it's going from schoolclass => teacher
# => schoolclass (same class) department is already populated
assert classes[0].students[0].schoolclass.name == "Math"
assert classes[0].students[0].schoolclass.department.name is None
await classes[0].students[0].schoolclass.department.load()
@ -118,8 +95,8 @@ async def test_model_multiple_instances_of_same_table_in_schema():
@pytest.mark.asyncio
async def test_right_tables_join():
async with database:
async with database.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
await create_data()
classes = await SchoolClass.objects.select_related(
["teachers__category", "students"]
@ -133,8 +110,8 @@ async def test_right_tables_join():
@pytest.mark.asyncio
async def test_multiple_reverse_related_objects():
async with database:
async with database.transaction(force_rollback=True):
async with base_ormar_config.database:
async with base_ormar_config.database.transaction(force_rollback=True):
await create_data()
classes = await SchoolClass.objects.select_related(
["teachers__category", "students__category"]