Prodream/ormar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP - Pydantic v2 support (#1238)

Browse Source 
* WIP

* WIP - make test_model_definition tests pass

* WIP - make test_model_methods pass

* WIP - make whole test suit at least run - failing 49/443 tests

* WIP fix part of the getting pydantic tests as types of fields are now kept in core schema and not on fieldsinfo

* WIP fix validation in update by creating individual fields validators, failing 36/443

* WIP fix __pydantic_extra__ in intializing model, fix test related to pydantic config checks, failing 32/442

* WIP - fix enum schema in model_json_schema, failing 31/442

* WIP - fix copying through model, fix setting pydantic fields on through, fix default config and inheriting from it, failing 26/442

* WIP fix tests checking pydantic schema, fix excluding parent fields, failing 21/442

* WIP some missed files

* WIP - fix validators inheritance and fix validators in generated pydantic, failing 17/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - working on proper populating __dict__ for relations for new schema dumping, some work on openapi docs, failing 13/442

* WIP - remove property fields as pydantic has now computed_field on its own, failing 9/442

* WIP - fixes in docs, failing 8/442

* WIP - fix tests for largebinary schema, wrapped bytes fields fail in pydantic, will be fixed in pydantic-core, remaining is circural schema for related models, failing 6/442

* WIP - fix to pk only models in schemas

* Getting test suites to pass (#1249)

* wip, fixing tests

* iteration, fixing some more tests

* iteration, fixing some more tests

* adhere to comments

* adhere to comments

* remove unnecessary dict call, re-add getattribute for testing

* todo for reverse relationship

* adhere to comments, remove prints

* solve circular refs

* all tests pass 🎉

* remove 3.7 from tests

* add lint and type check jobs

* reforat with ruff, fix jobs

* rename jobs

* fix imports

* fix evaluate in py3.8

* partially fix coverage

* fix coverage, add more tests

* fix test ids

* fix test ids

* fix lint, fix docs, make docs fully working scripts, add test docs job

* fix pyproject

* pin py ver in test docs

* change dir in test docs

* fix pydantic warning hack

* rm poetry call in test_docs

* switch to pathlib in test docs

* remove coverage req test docs

* fix type check tests, fix part of types

* fix/skip next part of types

* fix next part of types

* fix next part of types

* fix coverage

* fix coverage

* fix type (bit dirty 🤷)

* fix some code smells

* change pre-commit

* tweak workflows

* remove no root from tests

* switch to full python path by passing sys.executable

* some small refactor in new base model, one sample test, change makefile

* small refactors to reduce complexity of methods

* temp add tests for prs against pydantic_v2

* remove all references to __fields__

* remove all references to construct, deprecate the method and update model_construct to be in line with pydantic

* deprecate dict and add model_dump, todo switch to model_dict in calls

* fix tests

* change to union

* change to union

* change to model_dump and model_dump_json from dict and json deprecated methods, deprecate them in ormar too

* finish switching dict() -> model_dump()

* finish switching json() -> model_dump_json()

* remove fully pydantic_only

* switch to extra for payment card, change missed json calls

* fix coverage - no more warnings internal

* fix coverage - no more warnings internal - part 2

* split model_construct into own and pydantic parts

* split determine pydantic field type

* change to new field validators

* fix benchmarks, add codspeed instead of pytest-benchmark, add action and gh workflow

* restore pytest-benchmark

* remove codspeed

* pin pydantic version, restore codspeed

* change on push to pydantic_v2 to trigger first one

* Use lifespan function instead of event (#1259)

* check return types

* fix imports order, set warnings=False on json that passes the dict, fix unnecessary loop in one of the test

* remove references to model's meta as it's now ormar config, rename related methods too

* filter out pydantic serializer warnings

* remove choices leftovers

* remove leftovers after property_fields, keep only enough to exclude them in initialization

* add migration guide

* fix meta references

* downgrade databases for now

* Change line numbers in documentation (#1265)

* proofread and fix the docs, part 1

* proofread and fix the docs for models

* proofread and fix the docs for fields

* proofread and fix the docs for relations

* proofread and fix rest of the docs, add release notes for 0.20

* create tables in new docs src

* cleanup old deps, uncomment docs publish on tag

* fix import reorder

---------

Co-authored-by: TouwaStar <30479449+TouwaStar@users.noreply.github.com>
Co-authored-by: Goran Mekić <meka@tilda.center>
This commit is contained in:
collerek
2024-03-23 19:28:28 +01:00
committed by GitHub
parent 3a206dd8dc
commit 500625f0ec
294 changed files with 8132 additions and 9311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
docs/fastapi/index.md
View File

@ -16,17 +16,18 @@ Here you can find a very simple sample application code.
It's divided into subsections for clarity.
!!!note
If you want to read more on how you can use ormar models in fastapi requests and
responses check the [responses](response.md) and [requests](requests.md) documentation.
If you want to read more on how you can use ormar models in fastapi requests and
responses check the [responses](response.md) and [requests](requests.md) documentation.
## Quick Start
!!!note
Note that you can find the full quick start script in the [github](https://github.com/collerek/ormar) repo under examples.
Note that you can find the full quick start script in the [github](https://github.com/collerek/ormar) repo under examples.
### Imports and initialization
First take care of the imports and initialization
Define startup and shutdown procedures using FastAPI lifespan and use is in the
application.
```python
from typing import List, Optional
@ -36,29 +37,26 @@ from fastapi import FastAPI
import ormar
app = FastAPI()
metadata = sqlalchemy.MetaData()
database = databases.Database("sqlite:///test.db")
app.state.database = database
```
### Database connection
Next define startup and shutdown events (or use middleware)
- note that this is `databases` specific setting not the ormar one
```python
@app.on_event("startup")
async def startup() -> None:
database_ = app.state.database
if not database_.is_connected:
await database_.connect()
from contextlib import asynccontextmanager
from fastapi import FastAPI
@app.on_event("shutdown")
async def shutdown() -> None:
database_ = app.state.database
if database_.is_connected:
await database_.disconnect()
@asynccontextmanager
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
if not config.database.is_connected:
await config.database.connect()
yield
if config.database.is_connected:
await config.database.disconnect()
base_ormar_config = ormar.OrmarConfig(
metadata=sqlalchemy.MetaData(),
database=databases.Database("sqlite:///test.db"),
)
app = FastAPI(lifespan=lifespan(base_ormar_config))
```
!!!info
@ -71,21 +69,21 @@ Define ormar models with appropriate fields.
Those models will be used instead of pydantic ones.
```python
base_ormar_config = OrmarConfig(
metadata = metadata
database = database
)
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -122,7 +120,7 @@ async def create_category(category: Category):
@app.put("/items/{item_id}")
async def get_item(item_id: int, item: Item):
item_db = await Item.objects.get(pk=item_id)
return await item_db.update(**item.dict())
return await item_db.update(**item.model_dump())
@app.delete("/items/{item_id}")
@ -197,14 +195,14 @@ def test_all_endpoints():
assert items[0] == item
item.name = "New name"
response = client.put(f"/items/{item.pk}", json=item.dict())
assert response.json() == item.dict()
response = client.put(f"/items/{item.pk}", json=item.model_dump())
assert response.json() == item.model_dump()
response = client.get("/items/")
items = [Item(**item) for item in response.json()]
assert items[0].name == "New name"
response = client.delete(f"/items/{item.pk}", json=item.dict())
response = client.delete(f"/items/{item.pk}", json=item.model_dump())
assert response.json().get("deleted_rows", "__UNDEFINED__") != "__UNDEFINED__"
response = client.get("/items/")
items = response.json()

33
docs/fastapi/requests.md
View File

@ -23,11 +23,13 @@ Field is not required if (any/many/all) of following:
Example:
```python
base_ormar_config = ormar.OrmarConfig(
metadata=metadata
database=database
)
class User(ormar.Model):
class Meta:
tablename: str = "users"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
email: str = ormar.String(max_length=255)
@ -42,8 +44,8 @@ In above example fields `id` (is an `autoincrement` `Integer`), `first_name` ( h
If the field is nullable you don't have to include it in payload during creation as well as in response, so given example above you can:
!!!Warning
Note that although you do not have to pass the optional field, you still **can** do it.
And if someone will pass a value it will be used later unless you take measures to prevent it.
Note that although you do not have to pass the optional field, you still **can** do it.
And if someone will pass a value it will be used later unless you take measures to prevent it.
```python
# note that app is an FastApi app
@ -66,18 +68,18 @@ RequestUser = User.get_pydantic(exclude={"password": ..., "category": {"priority
@app.post("/users3/", response_model=User) # here you can also use both ormar/pydantic
async def create_user3(user: RequestUser): # use the generated model here
# note how now user is pydantic and not ormar Model so you need to convert
return await User(**user.dict()).save()
return await User(**user.model_dump()).save()
```
!!!Note
To see more examples and read more visit [get_pydantic](../models/methods.md#get_pydantic) part of the documentation.
To see more examples and read more visit [get_pydantic](../models/methods.md#get_pydantic) part of the documentation.
!!!Warning
The `get_pydantic` method generates all models in a tree of nested models according to an algorithm that allows to avoid loops in models (same algorithm that is used in `dict()`, `select_all()` etc.)
The `get_pydantic` method generates all models in a tree of nested models according to an algorithm that allows to avoid loops in models (same algorithm that is used in `model_dump()`, `select_all()` etc.)
That means that nested models won't have reference to parent model (by default ormar relation is bidirectional).
That means that nested models won't have reference to parent model (by default ormar relation is bidirectional).
Note also that if given model exists in a tree more than once it will be doubled in pydantic models (each occurrence will have separate own model). That way you can exclude/include different fields on different leafs of the tree.
Note also that if given model exists in a tree more than once it will be doubled in pydantic models (each occurrence will have separate own model). That way you can exclude/include different fields on different leafs of the tree.
#### Mypy and type checking
@ -94,7 +96,7 @@ RequestUser = User.get_pydantic(exclude={"password": ..., "category": {"priority
@app.post("/users3/", response_model=User)
async def create_user3(user: RequestUser): # type: ignore
# note how now user is not ormar Model so you need to convert
return await User(**user.dict()).save()
return await User(**user.model_dump()).save()
```
The second one is a little bit more hacky and utilizes a way in which fastapi extract function parameters.
@ -105,7 +107,7 @@ You can overwrite the `__annotations__` entry for given param.
RequestUser = User.get_pydantic(exclude={"password": ..., "category": {"priority"}})
# do not use the app decorator
async def create_user3(user: User): # use ormar model here
return await User(**user.dict()).save()
return await User(**user.model_dump()).save()
# overwrite the function annotations entry for user param with generated model
create_user3.__annotations__["user"] = RequestUser
# manually call app functions (app.get, app.post etc.) and pass your function reference
@ -126,8 +128,7 @@ Sample:
import pydantic
class UserCreate(pydantic.BaseModel):
class Config:
orm_mode = True
model_config = pydantic.ConfigDict(from_attributes=True)
email: str
first_name: str
@ -139,5 +140,5 @@ class UserCreate(pydantic.BaseModel):
async def create_user3(user: UserCreate): # use pydantic model here
# note how now request param is a pydantic model and not the ormar one
# so you need to parse/convert it to ormar before you can use database
return await User(**user.dict()).save()
return await User(**user.model_dump()).save()
```

70
docs/fastapi/response.md
View File

@ -22,11 +22,13 @@ Field is not required if (any/many/all) of following:
Example:
```python
base_ormar_config = ormar.OrmarConfig(
metadata=metadata
database=database
)
class User(ormar.Model):
class Meta:
tablename: str = "users"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
email: str = ormar.String(max_length=255)
@ -50,9 +52,9 @@ async def create_user(user: User): # here we use ormar.Model in request paramet
That means that if you do not pass i.e. `first_name` in request it will validate correctly (as field is optional), save in the database and return the saved record without this field (which will also pass validation).
!!!Note
Note that although you do not pass the **field value**, the **field itself** is still present in the `response_model` that means it **will be present in response data** and set to `None`.
Note that although you do not pass the **field value**, the **field itself** is still present in the `response_model` that means it **will be present in response data** and set to `None`.
If you want to fully exclude the field from the result read on.
If you want to fully exclude the field from the result read on.
### FastApi `response_model_exclude`
@ -61,7 +63,7 @@ Fastapi has `response_model_exclude` that accepts a set (or a list) of field nam
That has it's limitation as `ormar` and `pydantic` accepts also dictionaries in which you can set exclude/include columns also on nested models (more on this below)
!!!Warning
Note that you cannot exclude required fields when using `response_model` as it will fail during validation.
Note that you cannot exclude required fields when using `response_model` as it will fail during validation.
```python
@app.post("/users/", response_model=User, response_model_exclude={"password"})
@ -96,9 +98,9 @@ with client as client:
```
!!!Note
Note how in above example `password` field is fully gone from the response data.
Note how in above example `password` field is fully gone from the response data.
Note that you can use this method only for non-required fields.
Note that you can use this method only for non-required fields.
#### Nested models excludes
@ -111,13 +113,13 @@ One is a dictionary with nested fields that represents the model tree structure,
Assume for a second that our user's category is a separate model:
```python
class BaseMeta(ormar.ModelMeta):
metadata = metadata
database = database
base_ormar_config = ormar.OrmarConfig(
metadata=metadata
database=database
)
class Category(ormar.Model):
class Meta(BaseMeta):
tablename: str = "categories"
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=255)
@ -125,8 +127,7 @@ class Category(ormar.Model):
class User(ormar.Model):
class Meta(BaseMeta):
tablename: str = "users"
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
email: str = ormar.String(max_length=255)
@ -147,39 +148,39 @@ Note that you can go in deeper models with double underscore, and if you want to
In example `response_model_exclude={"category__priority", "category__other_field", category__nested_model__nested_model_field}` etc.
!!!Note
To read more about possible excludes and how to structure your exclude dictionary or set visit [fields](../queries/select-columns.md#fields) section of documentation
To read more about possible excludes and how to structure your exclude dictionary or set visit [fields](../queries/select-columns.md#fields) section of documentation
!!!Note
Note that apart from `response_model_exclude` parameter `fastapi` supports also other parameters inherited from `pydantic`.
All of them works also with ormar, but can have some nuances so best to read [dict](../models/methods.md#dict) part of the documentation.
Note that apart from `response_model_exclude` parameter `fastapi` supports also other parameters inherited from `pydantic`.
All of them works also with ormar, but can have some nuances so best to read [dict](../models/methods.md#dict) part of the documentation.
### Exclude in `Model.dict()`
### Exclude in `Model.model_dump()`
Alternatively you can just return a dict from `ormar.Model` and use .
Like this you can also set exclude/include as dict and exclude fields on nested models too.
!!!Warning
Not using a `response_model` will cause api documentation having no response example and schema since in theory response can have any format.
Not using a `response_model` will cause api documentation having no response example and schema since in theory response can have any format.
```python
@app.post("/users2/", response_model=User)
async def create_user2(user: User):
user = await user.save()
return user.dict(exclude={'password'})
# could be also something like return user.dict(exclude={'category': {'priority'}}) to exclude category priority
return user.model_dump(exclude={'password'})
# could be also something like return user.model_dump(exclude={'category': {'priority'}}) to exclude category priority
```
!!!Note
Note that above example will nullify the password field even if you pass it in request, but the **field will be still there** as it's part of the response schema, the value will be set to `None`.
Note that above example will nullify the password field even if you pass it in request, but the **field will be still there** as it's part of the response schema, the value will be set to `None`.
If you want to fully exclude the field with this approach simply don't use `response_model` and exclude in Model's dict()
If you want to fully exclude the field with this approach simply don't use `response_model` and exclude in Model's model_dump()
Alternatively you can just return a dict from ormar model.
Like this you can also set exclude/include as dict and exclude fields on nested models.
!!!Note
In theory you loose validation of response here but since you operate on `ormar.Models` the response data have already been validated after db query (as ormar model is pydantic model).
In theory you loose validation of response here but since you operate on `ormar.Models` the response data have already been validated after db query (as ormar model is pydantic model).
So if you skip `response_model` altogether you can do something like this:
@ -187,13 +188,13 @@ So if you skip `response_model` altogether you can do something like this:
@app.post("/users4/") # note no response_model
async def create_user4(user: User):
user = await user.save()
return user.dict(exclude={'last_name'})
return user.model_dump(exclude={'last_name'})
```
!!!Note
Note that when you skip the response_model you can now **exclude also required fields** as the response is no longer validated after being returned.
Note that when you skip the response_model you can now **exclude also required fields** as the response is no longer validated after being returned.
The cost of this solution is that you loose also api documentation as response schema in unknown from fastapi perspective.
The cost of this solution is that you loose also api documentation as response schema in unknown from fastapi perspective.
### Generate `pydantic` model from `ormar.Model`
@ -210,14 +211,14 @@ async def create_user3(user: User):
```
!!!Note
To see more examples and read more visit [get_pydantic](../models/methods.md#get_pydantic) part of the documentation.
To see more examples and read more visit [get_pydantic](../models/methods.md#get_pydantic) part of the documentation.
!!!Warning
The `get_pydantic` method generates all models in a tree of nested models according to an algorithm that allows to avoid loops in models (same algorithm that is used in `dict()`, `select_all()` etc.)
The `get_pydantic` method generates all models in a tree of nested models according to an algorithm that allows to avoid loops in models (same algorithm that is used in `model_dump()`, `select_all()` etc.)
That means that nested models won't have reference to parent model (by default ormar relation is bidirectional).
That means that nested models won't have reference to parent model (by default ormar relation is bidirectional).
Note also that if given model exists in a tree more than once it will be doubled in pydantic models (each occurrence will have separate own model). That way you can exclude/include different fields on different leafs of the tree.
Note also that if given model exists in a tree more than once it will be doubled in pydantic models (each occurrence will have separate own model). That way you can exclude/include different fields on different leafs of the tree.
### Separate `pydantic` model
@ -229,8 +230,7 @@ Sample:
import pydantic
class UserBase(pydantic.BaseModel):
class Config:
orm_mode = True
model_config = pydantic.ConfigDict(from_attributes=True)
email: str
first_name: str