GraphQL backed by OpenSearch

Why are these changes being introduced:

* we are moving to GraphQL as our sole endpoint
* we are using OpenSearch as our backend

Relevant ticket(s):

* https://mitlibraries.atlassian.net/browse/RDI-101

How does this address that need:

* Adds new v2 fields to GraphQL Records
* Deprecates fields by both documenting them as
  deprecated and which field to use instead, and also copies the data
  to the old field when possible
* Updates CI to run both OpenSearch and ElasticSearch versions of the
  tests

Document any side effects to this change:

note: filtering is mostly broken still at the Opensearch model level

note: Flipflip was used as it makes it slightly easier to flip states
in testing. I considered just using ENV but it was a bit clunky.

note: a few fields were unable to be deprecated cleanly. We could
consider renaming the new fields to allow for deprecation.

The tests need to be run twice until a better solution is identified or
we remove the elasticsearch version of the GraphQL (which will be in
the next few months). This is caused by our GraphQL implementation only
loading the config the first time it is used. The effect is that if an
elasticsearch test is run first, all the OpenSearch tests fail or vice
versa. The workaround was to name all of the tests as `graphqlv1` or
`graphqlv2` and using a filter to exclude running them. CI will run both
versions of the tests separately. Locally, developers should choose to
run whichever makes the most sense for their context. I wasn't able to
come up with a great way to default to one locally yet.

.github/workflows/ci.yml

@@ -25,9 +25,12 @@ jobs:
         run: |
           bundle config path vendor/bundle
           bundle install --jobs 4 --retry 3
-      - name: run tests
+      - name: run tests graphql elasticsearch
         run: |
-          bundle exec rails test
+          bundle exec rails test --exclude /graphqlv2/
+      - name: run tests graphql opensearch
+        run: |
+          bundle exec rails test --exclude /graphqlv1/
       - name: Coveralls
         uses: coverallsapp/github-action@v1.0.1
         with:

Gemfile

@@ -8,6 +8,7 @@ gem 'bootsnap', require: false
 gem 'devise'
 gem 'elasticsearch', '~>6.8'
 gem 'faraday_middleware-aws-sigv4'
+gem 'flipflop'
 gem 'graphql'
 gem 'jbuilder'
 gem 'jwt'

Gemfile.lock

@@ -152,6 +152,8 @@ GEM
       aws-sigv4 (~> 1.0)
       faraday (>= 1.8, < 2)
     ffi (1.15.5)
+    flipflop (2.6.0)
+      activesupport (>= 4.0)
     globalid (1.0.0)
       activesupport (>= 5.0)
     graphiql-rails (1.8.0)
@@ -350,6 +352,7 @@ DEPENDENCIES
   dotenv-rails
   elasticsearch (~> 6.8)
   faraday_middleware-aws-sigv4
+  flipflop
   graphiql-rails
   graphql
   jbuilder

app/graphql/types/query_type.rb

@@ -14,63 +14,145 @@ def ping
       argument :id, String, required: true
     end
 
-    def record_id(id:)
-      result = Retrieve.new.fetch(id, Timdex::EsClient)
-      result['hits']['hits'].first['_source']
-    rescue Elasticsearch::Transport::Transport::Errors::NotFound
-      raise GraphQL::ExecutionError, "Record '#{id}' not found"
+    if Flipflop.v2?
+
+      field :info, InfoType, null: false, description: 'Information about the current endpoint'
+
+      def info
+        i = Timdex::OSClient.info
+        Rails.logger.info(i)
+        i
+      end
     end
 
-    field :search, SearchType, null: false,
-                               description: 'Search for timdex records' do
-      argument :searchterm, String, required: true
-      argument :from, String, required: false, default_value: '0'
-
-      # applied facets
-      argument :content_type, String, required: false, default_value: nil
-      argument :contributors, [String], required: false, default_value: nil
-      argument :format, [String], required: false, default_value: nil
-      argument :languages, [String], required: false, default_value: nil
-      argument :literary_form, String, required: false, default_value: nil
-      argument :source, String, required: false, default_value: 'All'
-      argument :subjects, [String], required: false, default_value: nil
+    if Flipflop.v2?
+      def record_id(id:)
+        result = Retrieve.new.fetch(id, Timdex::OSClient)
+        result['hits']['hits'].first['_source']
+      rescue Elasticsearch::Transport::Transport::Errors::NotFound
+        raise GraphQL::ExecutionError, "Record '#{id}' not found"
+      end
+
+      field :search, SearchType, null: false,
+                                 description: 'Search for timdex records' do
+        argument :searchterm, String, required: false, default_value: nil
+        argument :title, String, required: false, default_value: nil
+        argument :from, String, required: false, default_value: '0'
+
+        # applied facets
+        argument :content_type, String, required: false, default_value: nil
+        argument :contributors, [String], required: false, default_value: nil
+        argument :format, [String], required: false, default_value: nil
+        argument :languages, [String], required: false, default_value: nil
+        argument :literary_form, String, required: false, default_value: nil
+        argument :source, String, required: false, default_value: 'All'
+        argument :subjects, [String], required: false, default_value: nil
+      end
+    else
+      def record_id(id:)
+        result = Retrieve.new.fetch(id, Timdex::EsClient)
+        result['hits']['hits'].first['_source']
+      rescue Elasticsearch::Transport::Transport::Errors::NotFound
+        raise GraphQL::ExecutionError, "Record '#{id}' not found"
+      end
+
+      field :search, SearchType, null: false,
+                                 description: 'Search for timdex records' do
+        argument :searchterm, String, required: true
+        argument :from, String, required: false, default_value: '0'
+
+        # applied facets
+        argument :content_type, String, required: false, default_value: nil
+        argument :contributors, [String], required: false, default_value: nil
+        argument :format, [String], required: false, default_value: nil
+        argument :languages, [String], required: false, default_value: nil
+        argument :literary_form, String, required: false, default_value: nil
+        argument :source, String, required: false, default_value: 'All'
+        argument :subjects, [String], required: false, default_value: nil
+      end
     end
 
-    def search(searchterm:, from:, **facets)
-      query = construct_query(searchterm, facets)
+    if Flipflop.v2?
+      def search(searchterm:, title:, from:, **facets)
+        query = construct_query(searchterm, title, facets)
+
+        results = Opensearch.new.search(from, query, Timdex::OSClient)
+
+        response = {}
+        response[:hits] = results['hits']['total']['value']
+        response[:records] = results['hits']['hits'].map { |x| x['_source'] }
+        response[:aggregations] = collapse_buckets(results['aggregations'])
+        response
+      end
+    else
+      def search(searchterm:, from:, **facets)
+        query = construct_query(searchterm, facets)
 
-      results = Search.new.search(from, query, Timdex::EsClient)
+        results = Search.new.search(from, query, Timdex::EsClient)
 
-      response = {}
-      response[:hits] = results['hits']['total']
-      response[:records] = results['hits']['hits'].map { |x| x['_source'] }
-      response[:aggregations] = collapse_buckets(results['aggregations'])
-      response
+        response = {}
+        response[:hits] = results['hits']['total']
+        response[:records] = results['hits']['hits'].map { |x| x['_source'] }
+        response[:aggregations] = collapse_buckets(results['aggregations'])
+        response
+      end
     end
 
-    def construct_query(searchterm, facets)
-      query = {}
-      query[:q] = searchterm
-      query[:content_format] = facets[:format]
-      query[:content_type] = facets[:content_type]
-      query[:contributor] = facets[:contributors]
-      query[:language] = facets[:languages]
-      query[:literary_form] = facets[:literary_form]
-      query[:source] = facets[:source] if facets[:source] != 'All'
-      query[:subject] = facets[:subjects]
-      query
+    if Flipflop.v2?
+      def construct_query(searchterm, title, facets)
+        query = {}
+        query[:q] = searchterm
+        query[:title] = title
+        query[:content_format] = facets[:format]
+        query[:content_type] = facets[:content_type]
+        query[:contributor] = facets[:contributors]
+        query[:language] = facets[:languages]
+        query[:literary_form] = facets[:literary_form]
+        query[:source] = facets[:source] if facets[:source] != 'All'
+        query[:subject] = facets[:subjects]
+        query
+      end
+    else
+      def construct_query(searchterm, facets)
+        query = {}
+        query[:q] = searchterm
+        query[:content_format] = facets[:format]
+        query[:content_type] = facets[:content_type]
+        query[:contributor] = facets[:contributors]
+        query[:language] = facets[:languages]
+        query[:literary_form] = facets[:literary_form]
+        query[:source] = facets[:source] if facets[:source] != 'All'
+        query[:subject] = facets[:subjects]
+        query
+      end
     end
 
-    def collapse_buckets(es_aggs)
-      {
-        content_format: es_aggs['content_format']['buckets'],
-        content_type: es_aggs['content_type']['buckets'],
-        contributors: es_aggs['contributors']['contributor_names']['buckets'],
-        languages: es_aggs['languages']['buckets'],
-        literary_form: es_aggs['literary_form']['buckets'],
-        source: es_aggs['source']['buckets'],
-        subjects: es_aggs['subjects']['buckets']
-      }
+    if Flipflop.v2?
+      def collapse_buckets(es_aggs)
+        {
+          contributors: es_aggs['contributors']['contributor_names']['buckets'],
+          source: es_aggs['source']['buckets'],
+          subjects: es_aggs['subjects']['subject_names']['buckets'],
+          languages: es_aggs['languages']['buckets'],
+          literary_form: es_aggs['literary_form']['buckets'],
+
+          content_format: es_aggs['content_format']['buckets'],
+          content_type: es_aggs['content_type']['buckets']
+
+        }
+      end
+    else
+      def collapse_buckets(es_aggs)
+        {
+          content_format: es_aggs['content_format']['buckets'],
+          content_type: es_aggs['content_type']['buckets'],
+          contributors: es_aggs['contributors']['contributor_names']['buckets'],
+          languages: es_aggs['languages']['buckets'],
+          literary_form: es_aggs['literary_form']['buckets'],
+          source: es_aggs['source']['buckets'],
+          subjects: es_aggs['subjects']['buckets']
+        }
+      end
     end
   end
 end