Some fun challenges I have encountered over my career.
The problem: We have over 50,000 articles from 12 different Rails models that need to be full text searchable. The goal is to allow the large data set to be used as a research tool for our users.
The solution: Using Elasticsearch we can call the data much faster than using normal active record. Additionally it is designed for full text searching. This makes writing the queries cleaner and maintainable.
Adding the elasticsearch model gem makes it so you can write ruby style queries and easy integration to keep your database and elastic cluster in sync.
Gemfile
gem 'elasticsearch-model', '~> 0.1.8'
For this project there were a multitude of different models to query.
Where you see foobar I am simply replacing a production word with an arbitrary one to get the point across.
Searchable is a Rails concern that is meant to be applied to any model that should be full text searchable.
Simply add include Searchable to the top of the model you want to search, then specify what
fields and relationships you want returned.
models/article.rb
class Article < ApplicationRecord include Searchable #### Allows Elasticsearch model gem to search specified fields def as_indexed_json(options={}) self.as_json( include: { categories: { only: [:article_id, :name] }, people: { methods: [:name], only: [:name] }, users: { only: :name } }) end end
The searchable concern sets up all of the shared code necessary to make a model searchable.
concerns/searchable.rb
module Searchable require 'elasticsearch/transport' require 'elasticsearch/rails/lograge' extend ActiveSupport::Concern included do #### Callbacks to keep the postgres and the elasticsearch database in sync include Elasticsearch::Model include Elasticsearch::Model::Callbacks after_commit { __elasticsearch__.index_document } #### Sets the index name for all environments including pull requests spun up on Heroku index_name ['pr', ApplicationController.get_pr_number_from_url, Rails.env, model_name.plural].join('_') #### Sets max records returned, if speed/memory becomes a problem remove and set a limit to returned results settings index: {max_result_window: 100000} end end
The GlobalSearch class is where all customization to ElasticSearch gems built in search.
The comments on each method should hopefully describe what each method does.
models/global_search.rb
class GlobalSearch < ApplicationRecord #### list of all models that are searchable GLOBAL_SEARCH_LIST = [ FooBar1, FooBar2, FooBar3, FooBar4 ] #### Set default styles for Elasticsearch highlighting def self.highlight { order: 'score', require_field_match: false, number_of_fragments: 1, fragment_size: 300, pre_tags: [''], post_tags: [''], fields: {:'*' => {}} } end #### Account for Models that do, or don't have a publish date def self.set_filters(entity_type='', start_date, end_date) if entity_type.present? && (entity_type == 'foobar_1' || entity_type == 'foobar_2') {} else { range: { published_at: { gte: start_date ? start_date.to_date : (Time.zone.now - 100.years), lte: end_date ? end_date.to_date : Time.zone.now } } } end end #### Get the count of the returned results def self.global_count_query(query, match_phrase, must_not_have_words, start_date, end_date) match_all = {match: {_all: query}} must_not = {match: {_all: must_not_have_words}} must_match_phrase = {match_phrase: {_all: match_phrase}} filters = set_filters(start_date, end_date) if match_phrase.present? { should: match_all, must: must_match_phrase, must_not: must_not, filter: filters } else { must: match_all, must_not: must_not, filter: filters } end end #### Search query for when you can't use the elasticsearch_model gem built in global search def self.search_query(query, match_phrase, must_not_have_words, start_date, end_date, entity_type) match_all = {match: {_all: query}} must_not = {match: {_all: must_not_have_words}} must_match_phrase = {match_phrase: {_all: match_phrase}} filters = set_filters(entity_type, start_date, end_date) #### Update query if match phrase is in the params if match_phrase.present? { should: match_all, must: must_match_phrase, must_not: must_not, filter: filters } else { must: match_all, must_not: must_not, filter: filters } end end #### Decide which models are queried def self.get_model_list(model) if model.present? case when model == 'foobar_1' FooBar1 when model == 'foobar_2' FooBar2 when model == 'foobar_3' FooBar3 when model == 'foobar_4' FooBar4 else GLOBAL_SEARCH_LIST end else GLOBAL_SEARCH_LIST end end end
The following Javascript file calls search controller actions using ajax to load the search results. One call is used to load the main search response, the others are used to find the number of results returned and will be displayed in the sidebar.
javascripts/load_search_response.coffee
ready = -> $params = $('#js-data-params') $search = $params.data('search') $model = $params.data('model') $page = $params.data('page') $end_date = $params.data('end-date') $start_date = $params.data('start-date') $match_phrase = $params.data('match-phrase') $must_not_have_words = $params.data('must-not-have-words') if $search || $model || $end_date || $start_date || $match_phrase || $must_not_have_words $send_params = { search: $search, model: $model, page: $page, end_date: $end_date, start_date: $start_date, match_phrase: $match_phrase, must_not_have_words: $must_not_have_words } $.ajax( url: '/load_search_response' type: 'POST' data: $send_params beforeSend: ( -> $('#js-loading-indicator').toggleClass('hide'); $('#js-loading-indicator').toggleClass('spinning'); ) ).success( (result)-> $('#js-loading-indicator').toggleClass('hide') $('#js-loading-indicator').toggleClass('spinning'); $("#js-results").html(result); ).error(-> $('#error-loading-message').removeClass('hide') console.log "Error retrieving search results please try refreshing the page." ) $.ajax( url: '/load_total_results_count' type: 'POST' data: $send_params ).success( (result)-> $('#js-load-total-results-count').html(result); ).error(-> console.log "Error retrieving sidebar result." ) $(document).ready(ready) $(document).on('page:load', ready)
The search controller takes in the users params and uses them to make the Elasticsearch queries.
Because the params are used repeatedly they are set in a before action called :set_common_search_params.
I am using the Elasticsearch model gem to make the calls easily to my cluster hosted on AWS.
controllers/search_controller.rb
class SearchController < ApplicationController before_action :set_common_search_params, only: [ :index, :load_search_response, :load_total_results_count, :load_foobar_count, ] def set_common_search_params authorize! :search, :all @query = params[:search] || '' @model = params[:model] @page = params[:page] @end_date = params[:end_date] @start_date = params[:start_date] @match_phrase = params[:match_phrase] @must_not_have_words = params[:must_not_have_words] end def index end def load_search_response es_response = Elasticsearch::Model.search( { query: { bool: GlobalSearch.search_query( @query, @match_phrase, @must_not_have_words, @start_date, @end_date, @model ) }, highlight: GlobalSearch.highlight }, GlobalSearch.get_model_list(@model) ) @response = es_response ? es_response.paginate(page: @page, per_page: ApplicationController::PAGE_COUNT) : [] render partial: 'search/load_search_response' end def load_total_results_count @total_results = Elasticsearch::Model.search( { query: { bool: GlobalSearch.global_count_query( @query, @match_phrase, @must_not_have_words, @start_date, @end_date ) } }, GlobalSearch::GLOBAL_SEARCH_LIST ).results.total render partial: 'search/sidebar_ajax/load_total_results_count' end end
We make the params available through the #js-data-params which acts as the container element.
This file also contains all of the ID's for the AJAX calls.
search/index.haml
#js-data-params.container{'data-search' => params[:search], 'data-model' => params[:model], 'data-page' => params[:page], 'data-end-date' => params[:end_date], 'data-start-date' => params[:start_date], 'data-match-phrase' => params[:match_phrase], 'data-must-not-have-words' => params[:must_not_have_words] } - if params[:search].present? || params[:model].present? || params[:end_date].present? || params[:start_date].present? || params[:match_phrase].present? || params[:must_not_have_words].present? .row .col-md-3 = render 'filters' %ul.list-group %a.js-categories-toggle.list-group-item.list-heading{'aria-controls' => 'categoriesCollapse', 'aria-expanded' => 'false', 'data-toggle' => 'collapse', href: '#categoriesCollapse'} Filters By Category #categoriesCollapse.collapse.in #js-load-total-results-count .col-md-9 #js-loading-indicator.box-wrap.text-center.hide .fa.fa-spinner.fa-spin #error-loading-message.box-wrap.text-center.hide Error loading results... try to refresh the page. #js-results - else = render 'common/no_results_found'
This file is what is returned from the main AJAX call loading the paginated response. I am simply rendering the search results based on what model type is returned. When the response is empty we simply display "No results found".
search/load_search_response.haml
- if @response.present? = render 'common/pagination', type: @response #results.box-wrap - @response.each do |result| = render "search/results/#{result._type.pluralize}", result: result - else = render 'common/no_results_found'
The problem: Admins will be posting articles with videos hosted on their Vimeo and YouTube channels.
The solution: Admins paste the url into a basic string input which will be persisted in the database. The video player will take the string and convert it to the right format based on which host it is coming from.
The form works just like any other rails string input form. Simply make sure the data is persisted.
articles/form.haml
.form-group = f.label :video_url = f.text_field :video_url, class: 'form-control', placeholder: 'Add video url here...' .help-block Paste a url from Vimeo or YouTube.
In your view you will want to have some form of container element here i'm using the responsive bootstrap iframe class.
I check to make sure the string is present before attempting to display the video player.
Then I call the embed method and pass in the video_url
articles/index.haml
- if @article.video_url.present? .embed-responsive.embed-responsive-16by9 = embed(@article.video_url)
This method checks to the string to see if the words Vimeo or YouTube are in it. If they are it formats the url to their specific guidelines then spits out an iframe with the desired setting and CSS class.
helpers/article_helper.rb
def embed(url) if url.include? 'youtube' youtube_id = url.split('=').last @embed_url = "//www.youtube.com/embed/#{youtube_id}" elsif url.include? 'vimeo' vimeo_id = url.split('/').last @embed_url = "//player.vimeo.com/video/#{vimeo_id}" end content_tag(:iframe, nil, src: @embed_url, allowfullscreen: '', class: 'embed-responsive-item') end
The problem: Users need to be able to style content within an article.
The solution: Redcarpet is the perfect gem that will allow for easy markdown integration.
Gemfile
gem 'redcarpet', '~> 3.3', '>= 3.3.4'
The markdown controller holds only one action called index which is rendered using ajax.
We will be making a little widget with three tabs. One that will hold the text area,
one tab will hold the preview, and the last tab will show a markdown cheat sheet.
For this to work you will need to add gem 'redcarpet' to your gemfile.
controllers/markdown_controller.rb
class MarkdownController < ApplicationController def index render 'index', locals: {text: params[:text]}, layout: false end end
The index page simply calls a helper method that handles the conversion logic.
markdown/index.haml
= markdown_to_html(text)
The markdown helper is where you will decide which markdown features you want to be available for the end user. This is also where the Redcarpet render is called which converts the persisted text from your database into markdown.
helpers/markdown_helper.rb
module MarkdownHelper def markdown_to_html(text) options = { filter_html: true, hard_wrap: true, link_attributes: { rel: 'nofollow', target: '_blank' }, space_after_headers: true } extensions = { autolink: true, superscript: true, no_images: true, no_styles: true, strikethrough: true, highlight: true, tables: true, fenced_code_blocks: true } renderer = Redcarpet::Render::HTML.new(options) markdown = Redcarpet::Markdown.new(renderer, extensions) markdown.render(text).html_safe end end
I placed it in a partial called editor so that it can be called in multiple places within the codebase.
articles/form.haml
= form_for @article do |f| = render 'markdown/editor', f: f
The editor file uses Bootstraps tab component with the three tabs stated earlier. There is nothing crazy going on in
this file but pay attention to the js- classes. Those will be getting used to make an AJAX call in the next file.
markdown/editor.haml
.js-markdown-container %ul.nav.nav-tabs %li.active %a{'aria-controls' => 'tab-edit', 'aria-expanded' => 'true', 'data-toggle' => 'tab', href: '#tab-edit', role: 'tab'} Edit %li %a.js-markdown-trigger{'aria-controls' => 'tab-preview', 'data-toggle' => 'tab', href: '#tab-preview', role: 'tab'} Preview %li %a{'aria-controls' => 'markdown-guide', 'data-toggle' => 'tab', href: '#markdown-guide', role: 'tab'} Guide .markdown.tab-content %span.js-api.hide{'data-url' => markdown_path} #tab-edit.tab-pane.fade.in.active{'aria-labelledby' => 'tab-edit', role: 'tabpanel'} .markdown-editor.form-group = f.text_area :body, class: 'form-control js-markdown-source', placeholder: 'body text...' #tab-preview.tab-pane.fade{'aria-labelledby' => 'tab-preview', role: 'tabpanel'} .panel.panel-default .panel-body .js-html-preview #markdown-guide.tab-pane.fade{'aria-labelledby' => 'markdown-guide', role: 'tabpanel'} = render 'markdown/guide'
This file makes the AJAX call when the user goes to the preview tab. It essentially gets the source input then calls the markdown index method, and finally returns the converted markdown in the new tab. This all happens virtually instantaneous.
javascripts/markdown.coffee
ready = -> MarkdownPreview = {} MarkdownPreview.showPreview = () -> $parentContainer = $(this).closest('.js-markdown-container') $url = $parentContainer.find('.js-api').attr('data-url') $source = $parentContainer.find('.js-markdown-source').val() $js_html_preview = $parentContainer.find('.js-html-preview') if $source.length == 0 $js_html_preview.html $source else $.ajax( type: 'POST' url: $url data: {text: $source} ).done (data) -> $js_html_preview.html data return return $('.js-markdown-trigger').on 'shown.bs.tab', MarkdownPreview.showPreview $(document).ready(ready) $(document).on('page:load', ready)
This is the last tab used as a markdown cheat sheet. If this looks familiar, that's because it is the same "GitHub Flavored" markdown! I moved it into to a nice tab for our users because they were all relatively new to markdown.
markdown/guide.haml
.panel.panel-default .panel-body .box-wrap %h3 HEADERS %hr %code # This is an h1 tag %code ## This is an h2 tag %code ###### This is an h6 tag .clearfix .box-wrap %h3 LISTS %hr %h4 Unordered %code * Item 1 %code * Item 2 %code.indented-list-item * Item 2a %code.indented-list-item * Item 2b .clearfix %h4 Ordered %code 1. Item 1 %code 2. Item 2 %code 3. Item 3 %code.indented-list-item * Item 3a %code.indented-list-item * Item 3b .clearfix .box-wrap %h3 EMPHASIS %hr %code *This text will be italic* %code _This will also be italic_ %code **This text will be bold** %code __This will also be bold__ %code *You **can** combine them* .clearfix .box-wrap %h3 BLOCKQUOTES %hr %code As Grace Hopper said: %code > I’ve always been more interested %code > in the future than in the past. .clearfix .box-wrap %h3 IMAGES %hr %code %code Format:  .clearfix .box-wrap %h3 LINKS %code http://github.com - automatic! %code [GitHub](http://github.com) .clearfix .box-wrap %h3 BACKSLASH ESCAPES %hr .help-block Markdown allows you to use backslash escapes to generate literal characters which would otherwise have special meaning in Markdown’s forma ing syntax. Markdown provides backslash escapes for the following characters: .clearfix %code \*literal asterisks\* %code \ backslash %code ` backtick %code * asterisk %code _ underscore %code {} curly braces %code [] square brackets %code () parentheses %code # hash mark %code + plus sign %code - minus sign (hyphen) %code . dot %code ! exclamation mark .clearfix
The problem: The application needs to send daily text messages to users at a designated time.
The solution: Using a scheduler I will call a rake task that sends texts out to users with scheduled background jobs.
This solution requires two gems to work. The first is Twilio which will require creating an account. The Twilio API is used to send text messages. It also allows for other phone integrations like calls which is nice if the system needs to expand its features. Delayed Jobs is the other gem that will be used to run all processes in the background and not clog up the application. Lastly I use Heroku scheduler to run the rake task at the beginning of every day.
Gemfile
gem 'twilio-ruby', '~> 5.7.2' gem 'delayed_job_active_record', '~> 4.1', '>= 4.1.1'
Start by making a new file in the initializers folder for the basic setup. Store your API keys in something like a .env file for development. Make sure whatever you use is included in the gitignore.
initializers/twilio.rb
Twilio.configure do |config| config.account_sid = ENV['ACCOUNT_SID'] config.auth_token = ENV['AUTH_TOKEN'] end
The next step is to create a new class that can be reused for all our texting needs. This class will be initialized with a message and the phone number of who will be receiving it. We then create a method called text that will take care of the actual sending.
services/twilio_text_messenger.rb
class TwilioTextMessenger attr_reader :message def initialize(cell, message) @cell = cell @message = message end def text client = Twilio::REST::Client.new client.messages.create({ from: ENV['ACCOUNT_NUMBER'], to: @cell, body: @message }) end end
This rake task is called once a day by using Heroku Scheduler. It loops through all of the users that have notifications turned on
and calls the send_daily_message method
lib/tasks/message_que.rake
namespace :message_que do desc 'Creates a daily message que' task create_jobs: :environment do @users = User.where(pause_notifications: false) @users.each do |user| user.send_daily_message end end end
The user model holds a list of messages that are randomly chosen and sent. There are two methods added here, one called
send_welcome_message and send_daily_message. Both of these methods use the
TwilioTextMessenger class and delayed jobs to send them in the background. The welcome message is sent
using an after_create hook.
models/user.rb
class User < ApplicationRecord after_create :send_welcome_message devise :database_authenticatable, :registerable, :recoverable, :rememberable, :trackable, :validatable validates :email, :support_level, :cell_phone, :start_time, :end_time, presence: true MESSAGES = [ "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "Lorem ipsum dolor sit amet, consectetur adipiscing elit." ] def send_welcome_message unless Rails.env.development? TwilioTextMessenger.new(self.cell_phone, "Welcome!").text end end handle_asynchronously :send_welcome_message def send_daily_message TwilioTextMessenger.new( self.cell_phone, MESSAGES.rand ).text.delay( run_at: rand( self.start_time..self.end_time ) ) end end