#230 AVO Directory. Cover & Profile images. Grid VS Table. Nokogiri for metadata

Hello, friends. Let's continue building a directory with avo. Now, uh, in the previous episode, we created categories. So categories can have, uh, many listings through category listings. And if we, uh, open a category, we have, uh, a view of all the associated listings, and we can go and attach listings that are not yet attached to this category. And if it is a listing, we can see the categories that the listing is attached to, to we have, uh, all these resources. So category, listing, category, and listing inside vo. And now we are going to continue building our directory. Let's first have some, uh, inspiration from, uh, different directories around the internet. So here I have a few directories, one, two, and three. And curiously, all of them look more or less, uh, the same. Uh, let's try opening a few links in them. So here you see I have, uh, a homepage. I have some, uh, t uh, listings. If you open a listing, you see the description of the listing, and you also see, uh, a screenshot and a link to go to the website and the avatar. Here we have, uh, a list of categories. We open a category, we see the listings inside the category. We can open one of the listings. Again, we see a screenshot and we see, uh, a description and link to visit. Going to the third directory, again, listings on the homepage. And, uh, I don't think they are kind of categorized. Yeah, here, they're not really, oh, yeah, there are like categories like health and so on. So here I have the health category. I have the listings. And, uh, again, if I open a listing here, we don't have screenshots of the listings, but we do have the name, the description, and the logo. So let's, uh, make our directory actually look a bit better. Uh, to start, we can actually make, uh, it so that we have the URL. And just based on the URL, we are going to try to find the title, the description, and maybe an image, uh, from, um, each, uh, website that have listed in our listings. Now, um, here, uh, to start, I also, uh, created some seed data. So here in seeds, uh, I've got, uh, some, uh, URLs that are RAs UI component libraries. And, uh, it's actually based on, uh, this, uh, blog post of mine. This blog post also, uh, kind of inspired me to create a directory because, uh, here I kind of list, uh, Ruby RAs component libraries, uh, in a blog post. But, uh, it make much more sense if it was, uh, a category inside, uh, uh, re's tools director, I think. So, uh, let's start with adding a feature to be able to get the additional data, uh, about the website just by listing. Its URL. And the easiest way to get, uh, uh, more data about the website is, uh, to go to the homepage, I think, and, uh, pause the homepage for information that we need. Uh, so let's imagine I, uh, type in the a new listing cprs.com, and we'll have a tool that will visit the homepage, and our tool will, uh, get the title of the page, the description of the page, uh, the favicon, it'll be the logo and maybe the open graph image. So we would get this from the meta text here, you see in the head, we have, uh, the title of the website, the meta description. We can even get the keywords, we can get the open graph image if there is one. So if I cope with this image here, you see it's, uh, kind of an image of the website that I can, uh, just extract from the HTML that I'm passing. I don't even have to use a screenshot generating tool. Let's try doing this. Let's try extracting the metadata of a website, and we can do it using the Gem NOCO Giri. So, uh, I've actually written a quick blog post about it recently. Uh, we can use NOCO Giri to visit a RL, get the, the core data, like the title, the, uh, favicon, the description, the open graph image, and let's do it for each of the links. So, uh, I will start by creating a new job. I will say, uh, rails generate, uh, job, uh, crawl, uh, crawler, uh, your L crawler. Let's then open our job, and I will, uh, basically copy all of this, uh, inside. So I'm going to use NOCO to open a URL, uh, pause it and, uh, get all this, uh, core data. And, uh, let's try doing it. I can do it in the console. So, rails console, uh, your crawler job dot perform. Now, let's say result equals your crawler job, perform now. And the URL will be, uh, listing last URL. And here we have the page title. We didn't find the, uh, fon, URL, uh, we did find the meta description, and we did not find the, the OpenGraph image, URL, uh, for rails cuts. So they don't, uh, have a favicon set in the same way. And, uh, they don't have an OpenGraph, uh, uh, image. That's fine. Let's, uh, uh, try for another listing, listing do first URL Here, you see, we already have the favicon, the open draft image. So here we have, uh, more data. Well, it doesn't, uh, mean that all the websites will have, uh, all the fields filled in, but this, there's a good start. So now we have added this job to extract, uh, this core data for the websites, and let's store them, uh, inside our listing. Now, in theory, we would want to maybe store more different, uh, attributes from crawling a website. And I don't wanna create a separate attribute for each, uh, of these. Maybe I want to just, uh, store them as Jason, and, uh, this way I can add more attributes, uh, to crawl whenever I want. So, let's, uh, create immigration rails, generate migration, add, uh, payload to a listing listings, uh, payload js O uh, now I'm not doing a JSON b uh, column, though. JSON B is, uh, better and more performant. I'm just doing JSON because, uh, I wanted to be more database agnostic. And, uh, as far as I know, JSOB is not, uh, uh, supported by all the databases, uh, uh, at the moment. Okay, let's, uh, now try running this, uh, ultra job on, uh, listing. And, uh, we will store this, uh, hash inside the listing payload. So, uh, let's, uh, make it based on the resource, uh, and we will have, uh, URL equals resource, URL. This will be the payload. And, uh, I will say resource update payload. Um, so this in theory, should, uh, update the payload attribute inside the, the listing with this, uh, hash, let's try running it on the one of our listings listing, because listing first, okay, let's now, uh, uh, run the job. We'll crawl a job, perform now, uh, based on this listing. Now we invoke the listing and you see inside the payload, V saved all this data. And if you say, listing payload, we have this, we can say, get, uh, uh, OG image URL, and he have, uh, the image URL, so it looks good. Now, how do we display this, uh, data in our application? And how can we make it so that we can, uh, run the job to, and, uh, crawl the website from our, uh, app? Uh, we're going to use AVO actions. So, uh, if we go to the AVO documentation, uh, actions creating an action, we're going to run a generated to create an action, uh, a will run rails generate AVO action, uh, we can name it, uh, uh, URL crawler, same as the job. And now if we go to the, our folder, we have this actions, URL crawler, um, uh, update. The name will be like, uh, uh, metadata from URL. Uh, and on the record, we are going to run URL Crawler job perform later on the record. Uh, now we need to add this action to our, uh, listings. So going to our, our resources listing, let's see how it's done in the docs, uh, we have to define actions, so dev actions, and here we will say, action avo actions your L crawler. Okay. And now I think if the refresh, uh, our page, uh, of our listings here, you see we have this action updated metadata from URL. Now let's, uh, comment it out. Okay. See, there are no actions. And then I enable this action, and we have it visible. Now, let's, uh, also display the payload of, uh, each of these URLs to see if, uh, they have, uh, uh, the metadata extracted already or not. Let's, uh, have a field, uh, metadata. Oh, payload as text. Okay? So you see we have payload for just fun of them. And let's run, uh, the job on a couple of other listings. So you see, I select a couple of listings. I will, uh, run the job. Uh, the job must be running here in the background. So if I refresh, now you see I, I have updated this data. Wonderful. We have added the feature of, uh, uh, just having a URL, and, uh, and now we have the action to visit the website and extract the core data. Marvelous. Okay, so now for some of our listings, we have the URL to the favicon, the URL to the open graph image. And let's try displaying them. If we can display them inside the, uh, listing page, or we can display them in the list of the listings, but actually other has this beautiful feature cover and profile photos. So imagine in the background, we can display the screenshot of the website or the open draft image. And here that will be the logo. Let's try using this cover and profile photos, uh, feature from avo. So I will cope with this for profile photo and, uh, go into our listing view. Oh, actually, let's, uh, save our changes. It's good to make, uh, saves during the progress. So, uh, we will say job to crawl URL and, uh, update listing metadata. Listing payload. Okay, let's continue. So now we are going to add the profile photo and, uh, for the index view, uh, well, it's gonna be nothing. We're not going to display it on the index view, I think. But, uh, we'll have, uh, by default, uh, I think slash uh, we have, uh, uh, dot png. We have some kind of png yes, if we have, uh, icon png, let's say slash icon png. And here in, uh, the actual, uh, uh, show page, we are going to display the profile photo. And the profile photo will need, uh, link. Uh, it'll be, let's see how it is in, uh, our app at the moment. So it'll be record payload, digg, uh, it'll be favicon, URL. Okay. Now, let's refresh this one. And wonderful. You see we have the favicon displayed. Let's also do it for the cover image. So we have the profile photo, we have the cover image somewhere. Okay, cover photo. Let's go with this. So, uh, cover photo, uh, the size. Let's make it, uh, yeah, let's start with medium visible on show and forms. Uh, so, uh, the default image is going to be the one that we have, uh, slash image icon png. It's the one that comes, comes with the new rails app, but you can change it to anything like not found or whatever. And here we will have record payload, dig, open draft image, URL. Now let's go and refresh the listing. Wow. You see, we have, uh, the open draft, uh, uh, image as the background and the logo as, uh, the avatar of the listing. Let's, uh, try, uh, updating UL metadata for more of them. Let's go to super ails. Okay, here you see and define metta. Dig for nil because, uh, uh, payload is not defined on this, uh, record. It's empty. Let's say record payload at Dick here and here if, uh, there is no payload. Okay? So now there's nothing, uh, if we can try display in the icon if, uh, nothing is found. So you see we have the, like placeholders. Let's go and, uh, visit the UL and update the metadata. I will, again, refresh the page and you see the, uh, and the background image are here. Wonderful. Let's try experimenting the different sizes of cover photos. Let's make it large. Yeah, it's, uh, large now. Okay, so this is basically using the cover and, uh, profile photos from our, and I think it's a really cool feature to have, uh, out of, uh, the box. Okay, let's go back to the list of our listings. And here you see, we have the URL, we have the payload. If you don't want to display the payload, uh, at least on the, uh, index page, let's hide it from index. So, uh, I will have hide on index. Okay, let's refresh. Okay. And we do want to display at least the, uh, favicon, like the avatar of, uh, the records. So let's, uh, display it. I will, uh, do it using the vo external image, uh, field type. So, uh, we would have a field, uh, uh, yes logo as external image, and we are going to use a block to define it. Uh, yeah. So we can also address, uh, Neil. And, uh, we are going to add a logo. And it's going to be, uh, let's see. We have the resource available here, so it'll be record, yeah, the record, record payload, fcon, URL. Okay, go into the list of our listings. And here you see, we see the avatars. Let's make them, uh, visible before the URL itself. Okay. And we see some of the avatars looks, uh, great. I think now, uh, a next step, uh, can be having a grid view and uh, uh, uh, table view. So a has this feature grid view where you have beautiful images, uh, uh, if u have images, uh, uh, a title and a description. Instead of, uh, having, uh, this kind of table view. And, uh, let's say I'll use, are going to be able to switch between table view and, uh, grid view. Uh, going to all these, uh, director of websites, uh, all of them have, uh, a grid view enabled. Okay, so let's add the grid view. Uh, before we do it, I'm going to save our changes. So, uh, uh, avo, uh, profile and cover photo. Okay, let's add the grid view. So again, I will, uh, copy the default example code going to our listing. And here I will enable grid. And we need to define, uh, cover URL. Now here, it gives the default example if we have, uh, uh, attachments, but at the moment we don't have attachments. We just have, uh, a URL, uh, title and, uh, body Now title can be the, um, let's see, where is it? Title can be the URL name, but uh, it can also be the payload page title. I think it's better. So I will say record payload. Do, uh, at dig page, title body will be record payload at dig, meta description. And the cover URL will also be coming from the payload. Uh, it can be either open graph image, or it can be, uh, the favicon. Let's try both. Let's go to our listing. Okay. Mismatch post, maybe, yeah, I shouldn't have copied all of it. Okay. And here, you see, now we have the table view by default, and we can switch to the grid view. And, uh, in the grid view, you see, we see the, uh, screenshots of the websites, uh, or the open draft images. Let's try using not the, uh, open draft image UL, but favicon. Now I think the open graph image URLs, uh, in our case look, uh, more interesting. So let's leave them. Okay. And that's how we have a grid view and, uh, uh, table view by default, if we go to a category, let's open one of the categories. You see the, we have the listing view. Let's make the grid view that default, uh, uh, view. So if we can make it as default view like this, let's go back to our listing resource. And yeah, I think next to the grid view settings, I will say default view type will be grid. So now going back to our app, I will refresh. And you see we have the grid view looks quite cool. Let's go back to our, uh, listings. And you see they are as a grid by default, but we can switch to table if we want to. So I think it's some good progress. Uh, we basically added the, an other action to visit the website and extract the metadata. So I'm going to now do it for all these listings. I have this background jobs happening. I will now refresh and you see it, uh, go to the favicons or avatars of all the websites. If I visit one of them, I see the cover image, I see the favicon, and I see some, uh, details about each of these records looks, uh, much better. Now, uh, as a final touch, you see we are displaying the payload, but maybe we want to display like the meta description and, uh, uh, title in, uh, separate lines. Let's try doing it. So, uh, I will have, uh, a field for, uh, each of them field, uh, uh, meta title, for example, it'll be as, uh, text, I think, or as HTML, I don't remember. That's, uh, I think yes as text, uh, label title do. And let's try having something. So you have the meta title. I think this label title doesn't fe maybe the, uh, GitHub al was hallucinating. And here we can have, uh, uh, that title from the payload. So I will have record payload page title. Let's refresh here. You see it is in separate, uh, line. And we can do the same with, uh, meta description field, uh, meta description. And here you see we have them in two different lines. So, uh, we don't need to display this, uh, payload that is not, uh, nicely user facing anymore. We see the, uh, fabricant from the payload. We see the OpenGraph image from the payload. We see the title and the description. Let's, uh, completely hide the payload. We don't need to display it anymore. And, uh, let's try another listing to our app. Let's, uh, go and create a new listing. So you say, here we have the placeholders. I will add URL box, for example, so your l box.com, add in a listing. I will go and update the metadata. Now, I'll refresh. And, uh, it didn't find the favicon, but it did find the, the screenshot, the title, and the description. So it looks good.