How to automatically create a Flop.Schema for all fields?

[egze]

I'm trying to add Flop to all my schemas, and I don't want to manually @derive it in every module. I tried to use __MODULE__.schema(:fields) but it doesn't work, because it's not available at compile time.

What else can I do?

[woylie]

First of all, I hope you know what you're doing, since allowing all schema fields by default for filtering and sorting can pose a security risk if the parameters you pass to the Flop query functions come from untrusted sources (HTML forms, JSON APIs, GraphQL APIs...) and you don't have additional validation. I would advice against this and recommend being explicit.

If you're still sure you want to do this, you can try to derive the protocol somewhere separate from the schema module. That way, the schema module may be compiled before, and you might be able to use __MODULE__.schema(:fields) in the protocol derivation. See https://hexdocs.pm/elixir/1.14.4/Protocol.html#derive/3. I haven't tried this, though.

[egze]

Thanks. I think I know what I'm doing 😄

We want to generate context modules with functions like this:

def list_movies(criteria \\ []) do
  # do flop stuff
end

So it is easy to filter records dynamically, paginate, sort, etc. Together with the option to only allow certain fields:

Movies.list_movies(filters: [name: "Die Hard 3", bad_field: 1], allowed_fields: [:name])

we can also accept unsafe user params safely, but still give developers the option to filter by anything. And we also use Flop.validate

[egze]

Just curious, have you ever thought about making a library based on Flop to create contexts with built-in filtering, pagination capability? Because it's always a lot of boilerplate to have it. Usually it involves reducing params into an Ecto.Query and it's the same for every context.

[egze]

So, in Phoenix when you create a new context, mix phx.gen.context Movies Movie movies title:string year:integer it will create a module like this:

defmodule MyApp.Movies do

  import Ecto.Query, warn: false
  alias MyApp.Repo

  alias MyApp.Movies.Movie

  def list_movies do
    Repo.all(Movie)
  end

  def get_movie!(id), do: Repo.get!(Movie, id)
end

And some create, update, delete functions.

But now you of course want to query movies by different fields, maybe by matching on exact title, year. So you maybe change the list_movies function to be something like this:

def list_movies(criteria \\ []) do
  base = from m in Movie, as: :movie

  query =
    criteria
    |> Enum.reduce(base, fn
      {:title, title}, query_acc ->
        where(query_acc, [movie: movie], movie.title == ^title)
      
      {:year, year}, query_acc ->
        where(query_acc, [movie: movie], movie.year == ^year)
    end)

  Repo.all(query)
end

Probably you move some logic to a different module, like MovieQueries. But then someone asks, wouldn't it be great to query by partially matching the title - and you go add more criteria options.

And then there's still the get_movie! function. What if we want to also be able to scope what we select, like get_movie!(1, user: user) (so, only load the movie if created by user). So you build the similar thing for get and also extend it all the time.

Wouldn't it be great to have something like this?

defmodule MyApp.Movies do

  use Flop.Context, schema: Orchestrator.Movies.Movie
end

And it generates the list_*, get_* functions with meta-programming? Then out of the box you could just do:

Movies.list_movies(filters: [%{field: :title, op: :==, value: "Something"}])

Movies.get_movie!(1, filters: [%{field: :year, op: :==, value: 2023}])

And anything else that Flop supports? You could have modes when the result is a classic list, or a full-flop mode, with the %Meta{}.

[woylie]

Right, while this would be possible, I think this kind of abstraction can break quickly. In the projects I'm involved in, I barely ever need plain CRUD functions. There's always some extra business logic involved. For example, I usually prefer to put authorization checks directly into the context functions. In other cases, I might need to add a default WHERE clause to a list function that should always be applied, no matter what the filter parameters passed to the function look like. Furthermore, I prefer to keep parameters set by the system separate from the ones passed by the user. So then you'd need to add options to Flop.Context to allow for escape hatches, hooks, custom queries and what not. In the end, I think it would be harder to work with than being explicit. It's one layer of abstraction too many for me. If I want to spare me some time writing out list functions and @derive tags, I can easily customize the Phoenix generator templates.

[egze]

That's fair. Do you mind if I take a stab at it and call it flop_phoenix_context? I have it implemented in our project and I think it's a pretty clean solution.

[woylie]

Sure, just mention that it's an unofficial extension. I think flop_context would be a better name.

[egze]

Thanks. Will do