Atheneum Development Log

Keep the past, for all intents and purposes, where it is.
- Okabe Rintarou, Steins;Gate.

The Research

Need

Mind this post’s date, it’s still the lockdown caused by the one and only COVID-19 and I had no projects on hand. So… Well it was time to build yet another app based on the Japanese Culture I’m a f*ckin weeb I get it okay don’t judge me. So this time I wanted to make something out of my comfort zone, as in something without using an existing API for it (unlike the last weeb app I built, yes I am talking about Konachan-Sama), so I decided to make something by scraping existing sites made for weebs.

Enlightenment

And hello there 😍! I thought about making a sleek Manga / Manhwa / Manhua app using Flutter! And started my hunt to look for a website that I could leach off pretty easily.

Enters Manganelo a website that has a non-existent robots.txt (which not only made my scraping super easy) and a wide range of comics to fulfill almost every other SFW-Believing-Weeb out there.

Wireframing

The entire landing page was crawling with so much information, it almost made me anxious. I did not feel the need to make a professionally laid out wireframe for the UI. I decided to cover the entire UI using 2 tabs and display everything that was shown on the landing page

The Grind Begins

Started as Fun

So I started with writing data models for every element that I would be displaying, it was something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class Popular {
final String img;
final String url;
final String name;
Popular({this.img, this.url, this.name});
}

class MostPopular extends Popular {
MostPopular({this.url, this.name}) : super(url: url, name: name);
final String url;
final String name;
}

class Latest extends Popular {
Latest({this.img, this.url, this.name})
: super(img: img, url: url, name: name);
final String img;
final String url;
final String name;
}

class Genre extends Popular {
Genre({this.url, this.name}) : super(url: url, name: name);
final String url;
final String name;
}

Half-way still looks like Fun

Looks pretty easy right? Wait the nightmare is about to get started! After this I had to write the scrapping part, ahaha it was fun, until it wasn’t. I will show only one method here, others are similar to bear with me:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import 'package:html/dom.dart';

import 'package:atheneum/models/popular.dart';

Future<List<Popular>> getPopulars(List<Element> elements) async {
List<Popular> populars = [];
elements.forEach((element) {
Element slideCaption = element.querySelector('.slide-caption > h3 > a');
var p = Popular(
img: element.querySelector('img').attributes['src'],
url: slideCaption.attributes['href'],
name: slideCaption.attributes['title']);
populars.add(p);
});
return populars;
}

This isn’t Fun anymore

And then there was the next part where had to collect data from everywhere and assign the values to the variables inside the class using a constructor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import 'package:atheneum/api/utils/latest.dart';
import 'package:atheneum/api/utils/most_popular.dart';
import 'package:atheneum/api/utils/popular.dart';
import 'package:atheneum/constants/urls.dart';
import 'package:atheneum/models/latest.dart';
import 'package:atheneum/models/most_popular.dart';
import 'package:atheneum/models/popular.dart';
import 'package:html/dom.dart';
import 'package:html/parser.dart';
import 'package:http/http.dart' as http;
import 'package:atheneum/models/genre.dart';
import 'package:atheneum/api/utils/genre.dart';

Future<Document> getHomePage() async {
http.Response response = await http.get(baseUrl);
Document document = parse(response.body);

return document;
}

class HomeData {
HomeData (Document document) {
init(document);

}
List<Popular> populars = [];
List<Latest> latests = [];
List<MostPopular> mostPopular = [];
List<Genre> genre = [];

Future<Map<String, dynamic>> init(Document document) async {

List popularElements = document.querySelectorAll('.owl-carousel > .item');
List latestElements = document.querySelectorAll('.doreamon > .itemupdate');
List mpElements = document.querySelectorAll('.xem-nhieu-item');
List genreElements = document.querySelectorAll('td');
this.populars = await getPopulars(popularElements);
this.latests = await getLatest(latestElements);
this.mostPopular = await getMostPopular(mpElements);
this.genre = await getGenre(genreElements);

return {
"populars": this.populars,
"latests": this.latests,
"mostPopular": this.mostPopular
};
}
}

Moment of truth

Now since I did not build any UI, it was time to use the data that we will be parsing. Enters DefaultTabController & TabBarView to the rescue! And we also make use of CustomScrollView

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
class Home extends StatefulWidget {
@override
_HomeState createState() => _HomeState();
}

class _HomeState extends State<Home> with AutomaticKeepAliveClientMixin<Home>{
@override
bool get wantKeepAlive => true;

HomeData home;
@override
void initState() {
getHomePage().then((document) {
setState(() {
home = HomeData(document);
});
});
super.initState();
}

@override
Widget build(BuildContext context) {
super.build(context);
return DefaultTabController(
length: 2,
child: Scaffold(
backgroundColor: colorBlack,
appBar: AppBar(
elevation: 0,
backgroundColor: colorBlack,
title: Text("Atheneum"),
actions: <Widget>[
IconButton(icon: Icon(Icons.search), onPressed: () {
Navigator.push(context, MaterialPageRoute(builder: (_) => SearchScreen()));
}),
IconButton(icon: Icon(Icons.settings), onPressed: () {}),
],
bottom: TabBar(tabs: [
Padding(
padding: const EdgeInsets.all(8.0),
child: Text("Home"),
),
Padding(
padding: const EdgeInsets.all(8.0),
child: Text("Miscellaneous"),
)
]),
),
drawer: SafeArea(
child: Drawer(),
),
body: home != null
? TabBarView(
children: [FirstPage(home: home), SecondPage(home: home)])
: Center(
child: CircularProgressIndicator(),
)));
}
}

Source Code

And here is the first screen of the TabBarView:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import 'package:atheneum/api/home.dart';
import 'package:atheneum/constants/color.dart';
import 'package:cached_network_image/cached_network_image.dart';
import 'package:flutter/material.dart';
import 'package:atheneum/screens/manga/mangascreen.dart';
import 'mangacard.dart';
import 'sliver_heading_text.dart';
import 'sliverdivider.dart';

class FirstPage extends StatelessWidget {
const FirstPage({
Key key,
@required this.home,
}) : super(key: key);

final HomeData home;

@override
Widget build(BuildContext context) {
return CustomScrollView(
slivers: <Widget>[
SliverHeadingText(
text: "New Popular Manga: ",
),
SliverToBoxAdapter(
child: Container(
height: MediaQuery.of(context).size.height * 0.3,
child: Stack(
children: <Widget>[
Positioned.fill(
child: ListView.builder(
scrollDirection: Axis.horizontal,
shrinkWrap: true,
itemCount: home.populars.length,
itemBuilder: (BuildContext context, int index) {
return InkWell(
splashColor: colorYellow,
onTap: () {
Navigator.push(
context,
MaterialPageRoute(
builder: (_) => MangaScreen(
url: home.populars[index].url,
img: CachedNetworkImage(imageUrl: home.populars[index].img),
)));
},
child: Padding(
padding: const EdgeInsets.all(4.0),
child: Card(
child: CachedNetworkImage(imageUrl: home.populars[index].img)),
),
);
},
)),
Positioned(
// right: -1,
child: Align(
alignment: Alignment.centerRight,
child: SizedBox(
width: 40,
child: FloatingActionButton(
backgroundColor: colorLight,
foregroundColor: colorBlue,
onPressed: () {},
child: Icon(Icons.arrow_forward),
),
),
))
],
),
...

Source Code

Fruit of the labor

Conclusion

Well the project was tiresome and I had to write alternative scraping mechanisms as the dumb website redirects some of the comics, still there were / are performance issues somewhere and I think they need to be taken care of but as of now the app will stay in it’s 2nd beta pre-release state. But, I will surely plan to update it overtime and hopefully it will turn out to be interesting.

One can find the Github Repository here