i Preface Welcome to the Volume 12 Number 1 of the International Journal of Design, Analysis and Tools for Integrated Circuits and Systems (IJDATICS). This volume is comprised of selected research papers from the International Conference on Recent Advancements in Computing in Artificial Intelligence, Internet of Things and Computer Engineering Technology (CICET), October 24-26, 2022, Taipei, Taiwan. CICET 2022 is hosted by The Tamkang University amid pleasant surroundings in Taipei, which is a delightful city for the conference and traveling around. CICET 2022 serves a communication platform for researchers and practitioners both from academia and industry in the areas of Computing in Artificial Intelligence (AI), Internet of Things (IoT), Integrated Circuits and Systems and Computer Engineering Technology. The main target of CICET 2022 is to bring together software/hardware engineering researchers, computer scientists, practitioners and people from industry and business to exchange theories, ideas, techniques and experiences related to all aspects of CICET. Recent progress in Deep Learning (DL) has unleashed some of the promises of AI, moving it from the realm of toy applications to a powerful tool that can be leveraged across a wide number of industries. In recognition of this, CICET 2022 has selected Artificial AI and Machine Learning (ML) as this year’s central theme. The Program Committee of CICET 2022 consists of more than 150 experts in the related fields of CICET both from academia and industry. CICET 2022 is organized by The Tamkang University, Taipei, Taiwan, and co-organized by AI University Research Centre (AI-URC) and Research Institute of Big Data Analytics (RIBDA), Xi’an Jiaotong-Liverpool University, China as well as supporting by: Swinburne University of Technology Sarawak Campus, Malaysia; Taiwanese Association for Artificial Intelligence, Taiwan; Trcuteco, Belgium; International Journal of Design, Analysis and Tools for Integrated Circuits and Systems, International DATICS Research Group. The CICET 2022 Technical Program includes 1 invited speaker and 30 oral presentations. We are beholden to all of the authors and speakers for their contributions to CICET 2022. On behalf of the program committee, we would like to welcome the delegates and their guests to CICET 2022. We hope that the delegates and guests will enjoy the conference. Professor Ka Lok Man, Xi’an Jiaotong-Liverpool University, China Professor Young B. Park, Dankook University, Korea Chairs of CICET 2022 ii Table of Contents Vol. 12, No. 1, January 2023 Preface ................................................................................................................................ i Table of Contents ............................................................................................................... ii 1. Jaeeun Kim, Hyonjun Kang, Mincheol Shin and Mucheol Kim, Implementation of Flood Hazard Area Construction with Dynamic Geofencing, Chung-Ang University, Seoul Korea 1 2. Swathi Anagondanahalli, Woon Kian Chong, Irina Pismennaya, Digital Fashion: Influence of K- Pop and go-to market strategy in the Metaverse, SP Jain School of Global Management, Singapore 4 3. Zhao-Yang Ni, Yu Su and Chii-Jen Chen, Electric Car Positioning to Prevent Car Accident Using a Smart APP Warning System, Tamkang University, Taiwan 10 4. Yuija Zhai and Thanasis Theodorou, Automatic Randomization in Assignment Question Design, University of Hertfordshire, UK 13 5. San-Hao Tsai, Shwu-Huey Yen and Hui-Jen Lin, Distilling One-Stage Object Detection Network via Decoupling Foreground and Background Features, Tamkang University, Taiwan 16 6. Wei-Chien Tai, Shwu-Huey Yen and Yihjia Tsai, Image Outpainting Based On Attention Model, Tamkang University, Taiwan 19 7. Wu Chengze, Wan Yamei, Li Wenbin, Kamran Siddique, An Overview of Adaptive Metadata Prefetching Scheme, Xiamen University Malaysia 26 8. Yuanyuan Luo, Chenxi Qin, Wenyu Zhang and Kamran Siddique, Design and Implementation of a Single-Cycle MIPS Processor, Xiamen University Malaysia 30 9. Shicheng Liao, Hua Zhong, Wenchong Wu, and Yungang Zhang, Lightweight Rotating Target Detection Model Based on Feature-Aligned Pyramidal Networks, Yunnan Normal University, China 35 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Implementation of Flood Hazard Area Construction with Dynamic Geofencing Jaeeun Kim, Hyonjun Kang, Mincheol Shin and Mucheol Kim Therefore, we implemented a geofencing constructing framework for information transmission service which Abstract: In modern society, urban inundation occurs transmit information to required area. Framework named frequently due to urbanization. Transmission of relevant dynamic flood hazard area construction uses historical flood information greatly affects flood risk awareness improvement hazard data to construct the geofence of flood hazard area and and can prevent flood damage. However, the current disaster information transmission system delivers information in update it dynamically. Concurrency is available with respect regional basis. Also, the area for information transmission to the nature of flooding that occurs simultaneously in a wide cannot be set up in real time. To solve this problem, we range and their interactions can be considered. Also, real-time implemented the dynamic flood hazard area construction operation is retained because it does not go through some framework, an area setting framework for the service that complex parameter operations. transmits information to the areas where flood related In summary, the contributions of this study are as follows: information is needed. The framework uses historical flood spatial data to establish flood risk areas and dynamically update • Implementation of geofencing framework for the it. It is possible to cope with flooding that occurs simultaneously transmission of information to area which flood hazard in a wide range. In addition, it is possible to transmit detailed exists. information by generating geofences step by step. Through the framework, it is possible to reduce flood damage by transmitting • Possible to respond to the interaction of flood that detailed information to the actual flooded areas and areas that occurs concurrently need information. Keywords: dynamic geofencing, flood risk, concurrent This paper is structured as follows: chapter 2 consists of execution released work, chapter 3 methodology, chapter 4 experiments, and chapter 5 conclusions. I. INTRODUCTION II. RELATED WORK In modern society, the social structure has been FLOOD RISK AWARENESS reorganized around cities [1]. Due to the population concentration by urbanization and the capacity limitation of Flood Risk The most common definition of flood hazard the drainage facility, urban inundation occurs frequently [2]. is the product of the hazard (e.g. intensity of damage by The lack of response to changes in rainfall by climate change flooding) and the vulnerability (e.g. probability of damage will intensify the urban inundation [3][4]. Since flooding is and weakness), means expected damage of specific incident unavoidable, it is necessary to cope with the damage with [12]. appropriate mitigation maneuver [5]. Flood Risk Awareness The components of flood risk Flood risk awareness of crowds mainly come from awareness are: (1) awareness of living in an at-risk area; relevant information and past experiences [6][7]. The low awareness of flood warning systems, codes and methods of level of preparedness due to the lack of information, wrong or dissemination; (3) awareness of appropriate action to take in delayed response are combined to cause hazards when the the event of a flood or flood warning [13]. actual flooding occurs [6][8]. Therefore, it is important to transmit relevant information in the flooding [5][9]. Methods such as awareness campaigns and risk However, the current disaster information transmission communication can be used to increase flood risk awareness. system has a problem in transmitting information on a However, since these methods are performed in a top-down regional basis. It can cause an imbalance in information manner, their effectiveness is largely unclear due to cultural transmission and problems that transmit information where differences or lack of local characteristics [6]. The agent- information is not required, or information is not transmitted based management (ABM) method, which views individuals to where information is required [10]. Moreover, the set up of as agents, is widely applied to cope with these problems [14]. the flooded area is often established through a questionnaire after a disaster or through a complex simulation model [11]. GEOFENCE Geofencing is a location-based service that creates a virtual All authors are with the Department of Computer Science and Engineering, fence (geofence) in a geographic area and automatically Chung-Ang University, Seoul, Republic of Korea detects when the tracked mobile objects enter or exit these (email: mindal99@cau.ac.kr, fender2758@cau.ac.kr, mcsin1648@gmail.com, areas. It can be applied to various fields related to the geo- kimm@cau.ac.kr). trackable objects [15]. With the generalization of mobile 1 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 devices, geofencing has become a common, popular service information related to flood hazard in the polygon update step. for many businesses especially in marketing and social media It can process multiple regions at once through concurrency [16]. and transmitting information about the relationship between polygons overlapping each other. Besides the static geofencing, dynamic geofencing which updates the geofence area in time is frequently utilized in the Polygon update Through the polygon update formula, the second phase zone can be stretched or contracted flexibly. The UAV (Unmanned Aerial Vehicles) fields. By flexibly formula uses the first phase zone and the former second phase reconstructing the movable area according to the movement zone to create a newly updated second phase zone. Since it is of UAVs, it can successfully perform their given missions by not smaller than the flood hazard map, it is possible to path planning or setting of no-go area of UAVs [17]. guarantee the area that requires minimal flood risk III. METHODOLOGY information transmission. Also, it can prevent an increase in computation during subsequent updated by preserving Chapter 3 introduces the dynamic flood hazard area polygon properties. construction framework. One cycle of the framework consists of two main steps: polygon construction and its update. In the IV. EXPERIMENTS first stage, a polygon of the first phase zone is constructed CASE STUDY using the flood hazard map. Then, perform a polygon update Busan is one of the most flood-prone cities in Korea. It is step using the first phase zone and the former second phase a large city with an area of about 769.89 km² and about 1.55 zone created in the previous cycle. In this way, the framework million inhabitants. It is located in the lower reaches of the that has finished one cycle creates a geofence step by step, and Nakdong River and is a port city facing the sea over a wide the result is used again for the next cycle. area. Busan is an area where a lot of rain is concentrated due The framework creates geofence areas in two stages to the rainy season between June and August and is frequently according to the expected flood risk level and the expected damaged by typhoons and heavy rain [18]. The reason why flood risk awareness. It is possible to provide more detailed Busan is selected is because it is flooded frequently by and realistic information through step differentiation. typhoons and heavy rains, also it is a large city in contact with rivers and seas. First phase zone The first phase zone is the area with the highest risk of flooding and is created through a polygon EXPERIMENT construction stage. The area is prone to experience flooding Flood Hazard Map Flood hazard map from the Korea directly, also residents have high flood risk awareness due to Land and Geospatial Informatix Corporation(LX) provided by past experiences and observations. the Open Government Data of the Korea was used. Second phase zone The second phase zone is the area which has an intermediate level of risk of flooding. Although it is unlikely to experience flooding directly, but residents should be aware of flooding and prepare inundation due to extreme flooding. This area changes due to the transmission of information through the movement of the residents of the first phase zone. In other words, it transmits flood risk awareness and serves as a buffer zone that prevents the influx of additional residents. POLYGON CONSTRUCTION In the polygon construction step, the most risk-prone area is defined by using a flood hazard map which has spatial distribution information of the past flood hazard. Due to the Fig. 1 The result of first phase zone construction. (A) is the raw flood hazard nature of flood damage, numerous hazardous area can appear map and (B) is the first phase zone constructed by convex hull operation. simultaneously over a wide range. It may be difficult to guarantee real-time performance due to the problem of First Phase Zone Construct a first -phase zone polygon computation time. Therefore, the convex hull operation is by convex hull operation on the flood hazard map. The raw performed based on the flood hazard map. flood hazard map of Busan has 391 flood hazard areas (in The amount of computation can be reduced by simplifying Figure 1 (A)). As a result of the convex hull, 124 of first phase the complex hazard area into simple convex polygons. zone polygons are constructed (in Figure 1 (B)). It shows that Therefore, real-time performance can be guaranteed by a valid flood risk area can be extracted through convex hull reducing the amount of computation. In addition, it is possible operation to create more robust area in the case of extreme flood than the past hazard area by convex polygons POLYGON UPDATE In this section, we construct a geofence that can cope with the varying flood hazard through dynamic area reconstruction via polygon update. Concurrency system Due to the nature of flooding which can occur simultaneously in a lot of hazardous areas in a wide region, there is a risk of real-time violation due to an increased Fig. 2 The result of second phase zone update. (A), (B), (C) depicts the update by time of the second phase polygon. calculation time. Therefore, the framework uses only 2 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Second Phase Zone Construct a dynamic geofence via [10] Bean, H., Sutton, J., Liu, B. F., Madden, S., Wood, M. polygon updates using the first phase zone and second phase M., & Mileti, D. S. (2015). The study of mobile public data from the previous cycle. It can be seen that the geofence warning messages: A research review and area dynamically changes with the passage of time (in Figure agenda. Review of Communication, 15(1), 60-80. 2). In addition, it can be seen that the geofence update is done [11] Merz, B., Thieken, A. H., & Gocht, M. (2007). Flood risk by reflecting the regional characteristics of the flood through mapping at the local scale: concepts and challenges. the change without losing the characteristics of the polygon. In Flood risk management in Europe (pp. 231-251). V. CONCLUSION Springer, Dordrecht. In this study, the dynamic flood hazard area construction [12] Apel, H., Aronica, G. T., Kreibich, H., & Thieken, A. H. framework was implemented for the transmission service to (2009). Flood risk analyses—how detailed do we need to provide flood information in preparation for urban inundation. be?. Natural hazards, 49(1), 79-98. Framework constructs hazardous areas of geofence using [13] Burningham, K., Fielding, J., & Thrush, D. (2008). ‘It'll flood hazard maps, which are historical flood area data, and never happen to me’: understanding public awareness of dynamically updates them. In addition, concurrency is local flood risk. Disasters, 32(2), 216-238. possible according to the characteristics of flooding and the [14] Zhuo, L., & Han, D. (2020). Agent-based modelling and interaction between hazard areas can be considered By flood risk management: a compendious literature reducing parameters, real-time operation can be possible and review. Journal of Hydrology, 591, 125600. detailed information about risk level can be transmitted by [15] Reclus, F., & Drouard, K. (2009, October). Geofencing constructing a geofence area step-by-step. Setting up a for fleet & freight management. In 2009 9th dynamic geofence through our framework, flood-related International Conference on Intelligent Transport information can be delivered to the actual flood damage area Systems Telecommunications,(ITST) (pp. 353-356). and areas requiring flood risk awareness, and further damage IEEE. can be reduced through appropriate damage response. [16] Suganya, V. (2022). USAGE AND PERCEPTION OF ACKNOWLEDGMENT GEOFENCING. EPRA International Journal of This research was supported by a grant(2021-MOIS37- Economics, Business and Management Studies 004) of Intelligent Technology Development Program on (EBMS), 9(2), 1-4. Disaster Response and Emergency Management funded by [17] Fabra, F., Vegni, A. M., Loscrí, V., Calafate, C. T., & Ministry of Interior and Safety(MOIS, Korea). Manzoni, P. (2022). Collision‐free cooperative Unmanned Aerial Vehicle protocols for sustainable REFERENCES aerial services. IET Smart Cities. [18] Ministry of the Interior ans Safety. (2022). A disaster [1] Brenner, N., Marcuse, P., & Mayer, M. (2009). Cities for yearbook. people, not for profit. City, 13(2-3), 176-184. [2] Prosdocimi, I., Kjeldsen, T. R., & Miller, J. D. (2015). Detection and attribution of urbanization effect on flood extremes using nonstationary flood‐frequency models. Water resources research, 51(6), 4244-4262. [3] IPCC. (2022). Climate Change 2022: Impacts, Adaptation and Vulnerability. [4] Waghwala, R. K., & Agnihotri, P. G. (2019). Flood risk assessment and resilience strategies for flood risk management: A case study of Surat City. International Journal of Disaster Risk Reduction, 40, 101155. [5] Mishra, A., Mukherjee, S., Merz, B., Singh, V. P., Wright, D. B., Villarini, G., ... & Stedinger, J. R. (2022). An overview of flood concepts, challenges, and future directions. Journal of hydrologic engineering, 27(6), 03122001. [6] Osberghaus, D., & Hinrichs, H. (2021). The effectiveness of a large‐scale flood risk awareness campaign: evidence from two panel data sets. Risk Analysis, 41(6), 944-957. [7] Tapsell, S. M., Penning-Rowsell, E. C., Tunstall, S. M., & Wilson, T. L. (2002). Vulnerability to flooding: health and social dimensions. Philosophical transactions of the royal society of London. Series A: Mathematical, Physical and Engineering Sciences, 360(1796), 1511- 1525. [8] Brilly, M., & Polic, M. (2005). Public perception of flood risks, flood forecasting and mitigation. Natural hazards and earth system sciences, 5(3), 345-355. [9] Lindell, M. K. (2013). Disaster studies. Current Sociology, 61(5-6), 797-825. 3 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Digital Fashion: Influence of K-Pop and Go- to Market Strategy in the Metaverse Swathi Anagondanahalli, Woon Kian Chong, Irina Pismeya Another landmark event that caught the world’s attention was when BTS, who lead the fame of K-pop, launched their song “Dynamite” Abstract— The Metaverse is a new world of opportunities and for the first time on a game called “Fortnite” on 25th September, 2020, limitless possibilities. The real-world impossibilities can be seen amidst the Covid-19 pandemic on the Metaverse platform. The official possible in the virtual world like international virtual concerts, music video on YouTube broke the record for being the biggest first- global launch of albums etc. We see celebrities engaging well with day debut with 101.1million views within 24 hours of its release. The their fans in this space and it cuts the noise from other song “Dynamite” being the BTS’s first-ever English language single, oversaturated platforms. K-Pop celebrities have been actively didn’t limit them to Asian viewership but has gained them immense involved in launching their unique collectibles, NFTs and digital popularity in the west too. [2] fashion. A huge influence of K-Pop stars has led to the increasing popularity and engagement of users on the Metaverse. Digital The study further will be based on the influential relationship of K- fashion on the Metaverse is found to be good match that brings Pop and what it has to offer to the Fashion retail industry and the Fashion to the world with innovation, that has not occurred in the Metaverse. As consumers are being more aware of “sustainable industry in a long time. Digital fashion could be the next goldmine fashion”, digital fashion could be the future with increasing awareness for fashion brands, as it’s the convergence of fashion with crypto among the GenZ and millennials currently. As Metaverse is a virtual technology for users with continuously evolving needs. As Fashion platform gaining popularity among niche sectors currently, the survey, plays a huge role for users to express their identities, “Avatars” is research and expert narrations carried out have helped determine the the new street wear and NFTs are the couture of the Metaverse. influence and the factors around it. While the experts of the industry have a positive take on this, the users currently don’t seem to accept the same. The space of digital fashion maybe hugely misunderstood by the majority for a long 2. LITERATURE REVIEW period from now, and so was the idea of being able to fly a plane Fashion marketing as a concept is still broadly misunderstood. In or drive a car, a few centuries ago. Until the majority of people the fast-fashion segment, it is perceived to be branding and selling accept, adopt and embrace these changes with a need that could through creating a demand for the creation of the designer. But in be fulfilled for them, the ambiguity will always exist around digital reality, the creation is influenced by the demand in the market, which fashion. is assessed by a fashion marketeer. A fashion marketeer should be well in sync with the designer to be able to get the right production. It’s the Index Terms— Digital fashion, Metaverse, NFTs, K-Pop, choice of making what can be sold over selling what can be made. [3] Avatars. As of March 2022, fashion retail is a $2.5 trillion industry [4]. Fashion 1. INTRODUCTION is a global business sector where trends are evolving and marketing plays a huge role to be able to support this change. In fashion, the aspects of design and marketing are always going hand-in -hand. The We have always seen examples of fashion labels aping the west in fashion industry runs on Planned obsolescence with trends getting fashion, but here was a classic example of the reverse. [1] Also, Dior exhausted and the demand for new creations is always high. [5] The has always marketed itself as a premium luxury brand that could be whole fast-fashion segment is monetized on the aspect of trends accessible to a very exclusive customer segment, but this move by the getting exhausted and consumption rising with the same. brand signifies inclusivity of a highly popular culture from the east and more target consumers with the growing fandom of K-Pop trend. By 2021, the boom in gaming industry also influenced the virtual fashion The generation Z (Genz) who are born after 1997, spend $44 billion as gamers dress up their “Avatars” with the latest styles, skins and also on digital assets such as smartphones, tablets, game consoles and integrating gaming with luxury fashion, resulting in premium gaming digital collectibles. This makes the generation well-connected socially merchandise. An iconic integration was when Gucci launched as the and highly active on the digital world. Marketing to GenZ and their first luxury fashion brand on the Metaverse with Roblox in May 2021, parents is the future of brands transforming digitally as they represent and created the Gucci Garden space on the virtual reality space. the market tomorrow. [6] Following this, was Ralph Lauren who launched on the Zepeto platform, introducing a range of Digital apparel for the 3D Avatars. To understand the influence of K-pop better, the search traffic was studied across the world and according to this study, the search traffic All authors are with the Department of Marketing, SP Jain School of Global in Korea was at its peak in the last quarter of 2020 and continued to Management, Singapore, (email: swathi.gm22ns028@spjain.org, increase in the same manner until March 2021, surpassing the global irina.pismennaya@spjain.org, tristan.chong@spjain.org). search traffic post June 2021. The popularity of Metaverse was high in Korea, compared to all other nations as majority of the search 4 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 phrases included “Metaverse” as a huge population in Korea among While professional gamers were some of the first consumers of the the GenZ have been avid gamers and further Metaverse stocks and Metaverse, fashion and entertainment industries have made their mark virtual currency used on the platform were found to be in Korea. This too in the virtual world. The study states that one in five Metaverse can indicate that consumers who were in their 30s or older too had users has been part of virtual concerts or film festivals. The most been involving in the Metaverse in the Korean region. Additionally, common interests of consumers on the Metaverse included playing when the Korean news was analysed, the changes in frequency were social games, immersive shopping and socializing. An average of similar to that of the search traffic studied. In the April of 2021, the $219 million has been spent on digital assets, out of which 30% is big news about the metaverse was when Zepeto, a metaverse platform Metaverse assets like real estate spaces, virtual enhancements and crossed 200 million users. By the May of 2021, fashion weeks, virtual collectibles like NFTs. The spike in popularity of luxury brands has fan signings, real estate meets had created the top news on the been a result of virtual lifestyle and luxury events which is attended by Metaverse. Researchers in 2010 had predicted that Metaverse would over half the people on the platform. [12] be a 3D immersive experience platform that would replace other e- commerce and web portals. Hence a good number of industries had a planned beginning on the platform as attention was given to the research. [7] The all-boy K-Pop band BTS is said to have a worldwide fan followership of ninety million. The fandom of K-Pop is significant as it influences various outcomes like social behaviours, technology adoption, psychological factors and being socially connected in their communities.[8] While we study the influence of K-Pop, a lot of idolizing occurs among fans which has been seen as a way of learning and upskilling themselves. The loyal fandom and the rise of women centered bands in the industry, has mad the fans to be positively well influenced by the stars. [9] A strong reason for the Korean industry to cross boundaries and huge success is the backing from their government. The Korean entertainment has a high factor of relatability to consumers and has gained huge viewership. With the rising popularity from the music and FIG 1: STUDY CONDUCTED IN UNDERSTANDING THE PARTICIPATION OF entertainment industries, the trend started to dictate on the fashion CONSUMERS IN VARIOUS IMMERSIVE DIGITAL ACTIVITIES OVER THE NEXT 5 trends across Korea and other Asian nations to start with. Currently in YEARS the west, K-Pop is considered a huge influencing trend as has huge fandom among millennials and GenZ. [10] The gamers on virtual platforms are drawn towards using fashionable gamification applications due to the financial incentives they receive The metaverse is a virtual platform where people are interacting in the like rewards, discounts and points. The scope for fashion labels in this form of “avatars” created and styled by them. Fashion in the metaverse segment of gamification applications is high as there are drawbacks platform plays a key role as users are in a 3-dimensional immersive like the usability. Availability of a user-friendly platform and looking environment. VR devices and smart phones have caused this to security and privacy concerns are the common areas of focus for aggressive reach of metaverse among the Gen Z. [11] fashion labels currently, addressing to the issues faced by consumers. [13] According to the McKinsey Global Fashion Index (MGFI), in 2019- 20, the fashion retail industry was vastly affected during the Covid-19 This study has huge significance as the US investment bank Morgan pandemic and witnessed a decline of 20% in revenues. This was a Stanley predicts that the digital fashion industry would reach over a trend that was applicable across other industries as well. A total of 69% $50 billion by 2030. With innovation into the virtual universe, a companies were termed to be value destroyers in 2020 and 7% of the fashion label can benefit very strongly financially with no physical companies left the market entirely, due to financial distress or production, distribution and labour costs and also earning royalties on acquisitions by competitors. It has been evident that labels who chose product resale. The issues of supply chain disruption during the Covid- to innovate into the digital platforms at the right time during the 19 pandemic would not play a role here as all merchandise is on the pandemic have been able to bounce back well. Fashion labels such as virtual universe. From the aspect of design and creativity, a world full Burberry, Louis Vuitton, Ralph Lauren and Gucci being a pioneer, of possibilities are open to create as the platform offers endless have started with stores on the metaverse. In the March of 2022, the opportunities. A worthy mention to this relationship is also the factor first ever metaverse fashion week was held on the virtual reality of sustainability. With digitalization, there is a huge saving on water platform Decentraland. This was one such landmark event in the and labour resources, less carbon emission and most importantly, history of fashion where over 500 digital garments were showcased waste management. The metaverse economy is thus predicted to be and over 70 well-known brands started with their stores in the worth between $8 -13 trillion by 2030 [14] metaverse during this digital fashion week. The reason this gained all the attraction is because neither the stores nor the products actually Marketing on the Metaverse platform is competing with that in the exist in reality and yet mostly were sold out. This form of physical world. While a huge number of consumes are drawn towards merchandising is a promising future of fashion and buyer habits. [4] the virtual platform out of curiosity, an effective engagement of consumers will have to be achieved. Platforms like Decentraland, Roblox, Fortnite are some of the pioneer and engaging platforms. 5 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 While fashion has never taken a back seat in a changing world, the easy to use. [20] In the current study, the adoption of Digital fashion luxury brand Gucci launched their metaverse edition of Gucci Garden on the Metaverse platform is understood better studying the factors on Roblox, which attracted 19.9 million consumers onto the platform that are affecting the behavioural factors of consumers. The perceived within two weeks of its launch. Gucci has been the pioneer luxury ease of use (using free of effort) and perceived usefulness (using the brand on the Metaverse to diversify across multiple games and technology enhances performance) are the factors under consideration platforms in the Metaverse, including Zepeto and blockchain-based to arrive at a theoretical conclusion. platform The Sandbox. Nike, one of the largest athletic apparel companies launched “Cryptokicks” which are unique NFTs with the virtual Nike dunk sneakers, designed by creative studio RTFKT. Fast- fashion brands like Forever 21 have started to make a mark in the Metaverse by selling a beanie for $1. Louis Vuitton has offered digital skins in the game League of Legends. The experience of virtual fashion using AR is allowing consumers to preview apparels, home décor items and instantly make a purchase of the physical world. Virtual objects are found to be the way consumers could experience a product or be part of the brand, which otherwise wouldn’t be accessible to them. [15] NFTs are the couture of the virtual fashion world. A large number of crypto enthusiasts have shown their interest in buying NFTs which are FIG 2: TECHNOLOGY ACCEPTANCE MODEL unique ownership of Digital art. Fashion labels have sought to launching NFTs to connect with gamers and crypto-enthusiasts, seeing Conceptual Framework: this as a strong tool of Metaverse marketing. NFT is a unique blockchain recording bound by a smart contract which is being As behavioural pattern of users is being studied, the following factors updated every time the NFT is being sold, traded or resold. The sale are considered to research and analyse their impact on the adoption of of an NFT allows the control and access of smart contract and Digital fashion on the Metaverse. practically any other utility of the same doesn’t exist. A huge benefit apart from the resale and value uniqueness, is the access to the Crypto communities. [16] The Metaverse life is perceived as a second life whereas the GenZ find the online and offline lives to be the same. Metaverse has been found to strengthen various factors like connectivity, virtual currency and better social activities. As there is growth in social activities and content on the Metaverse, the need for awareness exists which can help consumers adopt, beyond the GenZ. Currently the population on the Metaverse platform is very fluid and not consistent as involves even curious consumers who are willing to have a one-time immersive experience. The sustainability can be achieved by providing continual usability and keeping social relationships. [17] The need for digital fashion began with Gamers on the Metaverse who spent $100 billion in 2021 over virtual goods. Out of 3 billion people (over 30% of the world’s population) who are regularly playing video FIG 3: CONCEPTUAL FRAMEWORK games, 46% of these gamers are women and thus began the need for more fashionable virtual goods, with digital fashion. [18] 3. RESEARCH METHODOLOGY Among the many digitally native brands creating heat waves in the Digital fashion industry, RTFKT creates luxury NFTs, raised $8 Qualitative research is adopted for understanding the digital fashion million funding by a Silicon Valley venture capitalist firm in April segment and the growth of digital fashion on the Metaverse, by a 2021. Silicon Valley hardly invests in fashion and Andreessen narrative analysis in the form of interviews with industry experts. Horowitz, from the venture capitalist firm could foresee the potential Quantitative research is adopted to study the popular culture of K-Pop involved from the gaming perspective and how the industry is fast on fashion with various parameters considered in the survey and the evolving. [19] This implies how the industry revolution occurring in intersection of K-Pop, fashion and Metaverse is studied with human the world of fashion is drawing the attention of many, including the behaviour factors for digital fashion using Survey research. The Silicon Valley. sampling method opted for the above quantitative research will be Convenience Sampling. The research is expected to be carried out Technology Acceptance Model (TAM): among GenZ and millennials, who are the majority of existing and A cognitive framework to understand the adoption of a new potential consumers of Digital Fashion. technology can be achieved with Technology Acceptance Model. People (consumers) create conclusions and develop a bias prior to complete understanding of the technology and adapt to it when found 6 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 4. KEY FINDINGS Qualitative analysis: The influence of K-Pop is very strong and draws the attention of people to various platforms like live-streams, virtual reality events etc. This occurs more frequently in the regions of Asia, especially in China and Korea. The fandom of K-Pop is more intense across Asia who not only follow K-Pop music, but also their fashion, events and games. Linear regression analysis: Diane Wallinger, fashion designer at The Fabricant, says that as the culture of gaming is strong in Asia, users are more open to explore the Metaverse and adopt Digital fashion. In the western countries, fashion Hypothesis 1: Social norm is positively influencing the adoption of K- is considered more craftmanship oriented and importance is given to Pop with Digital Fashion on the Metaverse texture, feel and physical comfort. With the fast evolution of fashion brands on the Metaverse, increasing number of fashion labels are adopting Digital fashion. While technology has played a huge role in fashion over the last 5 years, virtual reality would be the next game changer for fashion. “Labels which are traditionally functioning will also have to pivot to digital fashion” says Ms.Wallinger. Digital fashion designs have no wastage and creation requires 97% lesser carbon-di-oxide. Brands are considering Metaverse expansion plans and soon physical fashion shows can be replaced by virtual events. As the space is interactive, the community of designers, creators and gamers is strong and growing, who encourage new creations. Simon Mikolajczyk, co-founder of Maison DAO, has been in the Digital fashion industry for over a decade and shared some insights. Mr.Mikolajczyk shares about the creative freedom from the angle of the designers. Designers of digital fashion experience no limitations in their designing horizons. The possibility of 3D-Designs and designs which defy the laws of gravity are an outstanding feature of Digital Hypothesis 2: The attitude of consumers is positive towards the fashion. As in physical fashion, the designs need not hang down (due acceptance of Digital fashion on the Metaverse to gravity) and defying these rules, digital designs can be created with absolute imagination. The real-world fashion challenges are tackled in the virtual world and the advantages of digital fashion over the traditional form are rising. While the debate can always go on about how digital fashion will never have that edge as in physical fashion, yet the experts have to conclude that digital fashion is here to stay. Quantitative analysis: Hypothesis 3: Technology Averse behaviour of consumers is stopping consumers to adopt Digital fashion on Metaverse 7 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 texture, value and wearability after sometime but all of this can be enhanced in the digital world. In my research, I learnt how existing fashion labels have expanded into the virtual reality space or plan to expand, with creations of sustainable fashion. As a practice, fashion brands opting the digital route have minimal wastage, minimal usage of resources (labour, energy, fabric) and overall, still high commercial revenue generation. The winners of the game are not just the fashion retailers who have expanded into the virtual space. This is a huge new opportunity for native digital retailers like The Fabricant, DressX to rise in fame. Technology companies have their share of benefit in this collaboration. 6. CONCLUSION The influence of K-Pop is one of the many strong driving factors to engage users on the Metaverse. Celebrities across Asia, especially from the Korean entertainment industry have been actively playing a role in the Metaverse and appears to be a match made in the revenue- Inferring from the quantitative results with survey and linear generating heaven. regression analysis, results are proving that the adopted hypotheses are wrong. As evidently for the 133 respondents, the results are: As a part of this research, K-Pop as a trend was studied to understand how the fandom is being used by fashion labels to capitalise on it. With • Social norm is not influencing people to adopt trends like K- the rise in virtual concerts, fan interactions, album launches of the K- Pop, Digital fashion Pop industry, there has been an increased number of K-Pop fans who • Users don’t have a positive attitude towards adopting Digital have been drawn towards the Metaverse. The luxury segment of fashion fashion has evolved to its full bloom by expanding on the virtual • Users are open to explore new technology changes and reality space. The driving factor for the luxury fashion labels to make hence are open to explore virtual reality platforms. a mark here is the success seen by digitally native brands that have evolved with the Metaverse like The Fabricant, DressX, and RTFKT. Hence the considered hypotheses have been proven wrong with A good competition exists between the brands who have also branched quantitative results. into the virtual world from the physical world and the digitally native brands. While the existing brands have a mark and recognition made Principal findings from Qualitative and Quantitative research: with the users earlier and the digitally native brands have the creative edge. [21] In comparison with earlier carried out literature review, the industry experts belonging to the fashion industry are of the opinion that K-Pop Users don’t see the need for digital fashion until they are regularly is one of the major factors of influence for users to adopt Digital involved in the immersive environment of the Metaverse. Hence a fashion. Digital fashion on the Metaverse is on a rise and here to stay market for Digital fashion exists with gamers, Meta-users and according to the experts. Therefore, we are seeing the rise of diverse attendees of virtual events. The value and usage of digital fashion may fashion labels on the Metaverse. With extreme possibilities and still be misunderstood by a majority who are yet to venture into the freedom of creation, digital fashion is growing with momentum. virtual world, while the experts of the fashion industry see this as the biggest revolution the fashion industry has ever seen. As a result of the The users of Digital fashion belong to a niche category, who are mostly survey conducted, hence users were not willing to adopt Digital gamers, fashion enthusiasts and Meta-users who are keen on building fashion and also found no relevance of it in their current life. [21] As their virtual identities/ Avatars. Apart from the usual Meta-users, an intersection of trends, K-Pop influencing the fashion on the adoption of Digital fashion by other users is not be preferred. Hence Metaverse with Digital fashion is currently evolving with a niche set as a result of the research survey that was carried out across a wide age of users but surely can dig gold for the fashion brands with time and group (GenZ and millennials included), under various factors of social apt marketing strategies. influence, attitude of people and technological barriers, the adoption of Digital fashion was not positively accepted. REFERENCES 5. IMPLICATIONS 1. CNN, O. H. (May 2022). Dior hosts runway show in South Korea for the first time. CNN. https://edition.cnn.com/style/article/dior- The real-world fashion is fabric-centered while digital fashion is more fall-2022-south-korea-show/index.html creative-centered. The limitations of fabric abilities, weight and 2. Mamo, H., & Mamo, H. (2020, September 21). BTS Set to Blow gravity don’t apply on the virtual world. This creativity in digital Up Fortnite With Brand New “Dynamite” Music Video. fashion is what makes the user’s experience more immersive, Billboard. https://www.billboard.com/music/music-news/bts- interactive and likeable for repetitive users. A piece of digital fashion fortnite-brand-new-dynamite-music-video-9453108/ once used, doesn’t lose value and can appreciate in value. It remains 3. Hines, T., & Bruce, M. (2007). Fashion Marketing: as a piece of art and can always be evolved. A physical garment loses Contemporary Issues. In Google Books. Routledge. 8 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 https://books.google.com.sg/books?hl=en&lr=&id=kPqa_Jj4wo 2022, from https://finance.yahoo.com/news/nft-brand-rtfkt- YC&oi=fnd&pg=PR3&dq=fashion+marketing+strategies&ots= raises-8m-232647190.html fbEYyIh9yW&sig=zsNVANTM6wNuSI5mlApDScwhb_0&red 20. Ajibade, P. (2018). Technology Acceptance Model Limitations ir_esc=y#v=onepage&q=fashion%20marketing%20strategies&f and Criticisms: Exploring the Practical Applications and Use in =false Technology-related Studies, Mixed-method, and Qualitative 4. Marketing in the metaverse: An opportunity for innovation and Researches. https://core.ac.uk/download/pdf/189486068.pdf experimentation | McKinsey. (2022). Www.mckinsey.com. 21. Lee, D., & Malik, R. (2021, November 3). The Opportunity in https://www.mckinsey.com/business-functions/growth- Digital Fashion and Avatars Report — BoF Insights. The marketing-and-sales/our-insights/marketing-in-the-metaverse- Business of Fashion. an-opportunity-for-innovation-and-experimentation https://www.businessoffashion.com/reports/technology/the- 5. Easey, M. (2009). Fashion Marketing. In Google Books. John opportunity-in-digital-fashion-and-avatars-report-bof-insights/ Wiley & Sons. 22. Gonzalo, A., Harreis, H., Sanchez Altable, C., & Villepelet, C. https://books.google.com.sg/books?hl=en&lr=&id=7KyIbhpkjG (2020, May 6). The fashion industry’s digital transformation: IC&oi=fnd&pg=PR5&dq=fashion+marketing&ots=HPxyLGp2 Now or never | McKinsey. McKinsey & Company. Ku&sig=aJIr7cWGriUeU6rW9wmklZncesc&redir_esc=y#v=on https://www.mckinsey.com/industries/retail/our- epage&q=fashion%20marketing&f=false insights/fashions-digital-transformation-now-or-never 6. Kotler, P., & Armstrong, G. (2018). Principles of Marketing (17th ed.). Pearson Education Limited. 7. Jee, Y., & Lee. (2021). A Study on Metaverse Hype for Sustainable Growth. International Journal of Advanced Smart Convergence, 10(3), 72–80. https://doi.org/10.7236/IJASC.2021.10.3.72 8. Laffan, D. A. (2020). Positive Psychosocial Outcomes and Fanship in K-Pop Fans: A Social Identity Theory Perspective. Psychological Reports, 124(5), 003329412096152. https://doi.org/10.1177/0033294120961524 9. Ding, Y., & Zhuang, X. (2021). Why Chasing Kpop? Is Fandom Truely Crazy? --The Motivations and Behaviors of Kpop Fans. https://doi.org/10.25236/ermss.2021.008 10. Roy Rishav Kumar, T., & Suraj. (2018). The Influence of Korean Pop Culture in East and Southeast Asian Nations. In International Journal of Advance Research and Development. https://www.ijarnd.com/manuscripts/v3i1/V3I1-1220.pdf 11. Mckinsey. (2021). The State of Fashion 2021. https://www.mckinsey.com/~/media/McKinsey/Industries/Retail /Our%20Insights/State%20of%20fashion/2021/The-State-of- Fashion-2021-vF.pdf 12. Myths of the metaverse: Facts vs fiction | McKinsey. (2022.). Www.mckinsey.com. https://www.mckinsey.com/industries/retail/our- insights/probing-reality-and-myth-in-the-metaverse 13. Roland Goldberg, Valeska Nel (2022). To Play or Not to Play: The Use of Gamification in The Fashion Retail Industry. Malaysian E Commerce Journal, 6(2): 34-38. 14. Metaverse and Money: Decrypting the Future. (March 2022). Icg.citi.com. https://icg.citi.com/icghome/what-we- think/citigps/insights/metaverse-and-money_20220330 15. Papagiannis, H. (2020). Harvard Business Review How AR Is Redefining Retail in the Pandemic. https://ismguide.com/wp- content/uploads/2022/03/How-AR-is-Redefining-Retail.pdf 16. Murray, M. D. (2022). Trademarks, NFTs, and the Law of the Metaverse. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4160233 17. Park, S.-M., & Kim, Y.-G. (2022). A Metaverse: taxonomy, components, applications, and open challenges. IEEE Access, 10, 1–1. https://doi.org/10.1109/access.2021.3140175 18. Nast, C. (2022, May 25). The digital designers making millions from in-game fashion. Vogue Business. https://www.voguebusiness.com/technology/the-digital- designers-making-millions-from-in-game-fashion 19. NFT Brand RTFKT Raises $8M, Plans to Build Digital Marketplace. (n.d.). Finance.yahoo.com. Retrieved October 2, 9 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Electric car positioning to prevent car accident using a smart APP warning system 1 Zhao-Yang Ni, 2Yu Su, and 1Chii-Jen Chen* Abstract— The electric car is already a global trend, however, be the first one to realize it while the Intelligent Transportation various car brands, various self-driving platforms, various cell System and Telematics provide the basic criteria for IOT based phone navigation, various systems are not connected to each other. The study hopes to use the simplest 5G network GPS Wi-Fi vehicle development [3]. Therefore, this application can be Bluetooth positioning system and even using Starlink, the applied to the car network. construction of a small 5G base station Wi-Fi, Bluetooth or After receiving the data from 5G communication system (or Starlink fixed transmitter at the road intersection pole to provide Wi-Fi, GPS, BeiDou, StarLink and Bluetooth), the data is positioning to avoid vehicle collisions and correct positioning gaps, to data transmission. Then the AI intelligent calculation of big data, stored in the cloud, and the AI intelligent management, cloud cloud storage, cloud computing, intelligent voice alert and control, computing, intelligent voice alert and control and the principle provides the positioning of each vehicle on the screen display to of IOT are used to calculate the distance and location of each remind, reduce that the electric car can’t identify the parked vehicle to provide driving reference, vehicle positioning and construction vehicles and electric car monitoring camera can’t road conditions to avoid car accidents. The display screen of identify the accident. various electric vehicles and car self-driving systems, including Index Terms— GPS, Car accident prevention, smart computation. mobile phone screen, shows the relative distance, speed and movement of each car, and finally achieves intelligent traffic control of speed and car control of fully automatic driving. I. INTRODUCTION II. THE PROPOSED METHOD GPS is a satellite-based positioning and navigation of a 2.1 The principle of positioning (upload) communication system. In general GPS research is focused on In Fig. 1, this proposed system provides vehicle information positioning, there is less research on signal and communication in each paragraph. Vehicle positioning includes height, vehicle systems [1]. Intelligent cruise control system is one of the main speed, vehicle steering, vehicle safety, vehicle collision, development directions of vehicle industry. Adaptive cruise network alarm and other situations. control (ACC) has been successfully applied to commercially available. It could reduce the burden on drivers when driving in (1) (2) (3) (4) (5) (6) a long time, but the function of ACC is limited to high-speed (7) (8) (9) (10) (11) (12) cruise use [2]. However, ACC can only detect the vehicle in front of you within one kilometer, beyond one kilometer will not be detected. If the vehicle in front of you is a car accident Corresponding Number: or engineering vehicles, ACC will not be able to detect. If we (1) User ID use GPS as the base, we can propose a reliable system that can (2) Longitude and latitude detect the road conditions three kilometers or even five (3) Speed kilometers ahead. (4) Direction: left and right After the cloud computing, the Internet of Things (IOT) has (5) Vehicle stopped got most of the attention on the global agenda. Not only the (6) Vehicle under construction leading companies have invested in R & D resources on it, but (7) Vehicle crashed also countries or regions have announced the policies to (8) 7 airbags opened incubate the IOT. The trend of vehicle intellectualization has (9) Vehicle temperature become increasingly obvious. The applications of Internet of Things have unlimited imagination. The IOT based vehicle may (10) Vehicle overturned (11) Original customer service (12) Network alarm The author's institution is as follows: Fig. 1. The screen view of proposed system. 1 Department of Computer Science and Information Engineering, Tamkang University, New Taipei City, Taiwan 2 Department of Medical Imaging and Radiological Technology, Yuanpei 2.2 Relative position (download) University of Medical Technology, Hsinchu, Taiwan Fig. 2 shows vehicle positioning including height, vehicle (email: eilotcraft429@gmail.com, suyu96@mail.ypu.edu.tw, speed, vehicle direction by each paragraph, provide vehicle cjchen@mail.tku.edu.tw). status within a 5 or 10 km radius with the positioning of the 10 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 vehicle, and provide hazard warning when the vehicle behind Icon Description exceeds the speed limit by more than 30 km/h in the same lane. The center of main car shows the relative position and speed of each vehicle, with various colors or patterns. 25.176 100km L The upper right corner is the vehicle stop or 121.440 construction or warning the vehicle behind to 25.376 100km R pay attention to safety. 121.480 The upper left corner is the vehicle accident, Xxx 0km S provide network alarm, provide the location of xxx the vehicle, vehicle safety, whether the vehicle is overturned, airbag burst several, provide the Xxx 20km C driver contact method, provide emergency xxx rescue. Xxx 140km W Function Setting xxx Zoom in Fig. 2. The coordinate screen of proposed system. Zoom out 2.3 Display principle The simulated map interface is shown in Fig. 3 and its functional icons descripted in Table 1. When the APP completed it will show like Fig. 4. The longitude and latitude unit of the screen display method is 1.3 meters. The longitude and latitude of the vehicle is the center and the hidden display points are set up at intervals of 5 square kilometers. The relative position of the vehicle and other vehicles is marked by the longitude and latitude. The satellite positioning uses mathematical programs. [(5000÷1.3)2 Unit x=1.3 meters] Wi-Fi, Bluetooth or Starlink positioning within a few centimeters, displayed on the screen, allowing drivers to judge the vehicle condition, or cooperate with the intelligent automatic driving system of electric vehicles to detect road conditions in advance, and adjust the safe relative position of each vehicle, and can also cooperate with the longitude and latitude of the navigation system to mark The relative position of the car and other vehicles to avoid accidents. Fig. 4. Delicate Map Interface simulation. (Note Citing Google Maps, simulated Map by completed APP Simulation.) 2.4 The principle of warning We listed four examples to describe the principle of warning: (1) Speeding vehicles, speed exceeds the car more than 30km display warning, as shown in Fig. 5. For example: direction XXX.XXX speed 100KM, the speed of the vehicle behind exceeds the speed of the car more than 30km, be careful. (2) Red warning vehicle in front, with the vehicle camera to change lanes early, as shown in Fig. 6. For example: direction Fig. 3. The map interface of proposed system. Table 1. The corresponding icons about Fig. 3. 11 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 XXX.XXX Speed 100KM, 1km with engineering vehicles. Please take turn. (3) The ambulance or firetruck from behind, then the vehicle give way, as shown in Fig. 7. For example: ambulances and fire trucks in the back, the car in front please give way. (4) The police car on duty from behind, then the vehicle make way, as shown in Fig. 8. For example: police car in the rear on official business, the car in front please give way. 100 km/h Fig. 8. An example of police car from behind. III. DISCUSSIONS The difference between ACC and my program is that ACC 140 km/h can only detect the moving vehicle within one kilometer ahead, while my program can detect the vehicle within three to five kilometers in advance by using GPS. ACC can’t detect the Fig. 5. An example of speeding car from behind. stopped vehicle ahead, if the vehicle ahead is a car accident or a construction vehicle, ACC cannot recognize it, my program can first detect the vehicle ahead whether it is a car accident vehicle or engineering vehicle. Furthermore, for the examples of warning cases in Section 2.4, the ACC system may can handle the warning cases in Fig. 0km/h 5 and Fig. 6. Nevertheless, the detection of vehicles coming from behind (in Fig. 7 and Fig. 8) may not be able to achieve the function of reminding the yield lane with the current ACC technology, but this function is the major advantage of our proposed system. IV. CONCLUSIONS 100km/h The program can be touch, with a variety of navigation map Fig. 6. An example of stopped car in front. directly on the display, can provide the road ahead can be changed early navigation, can also be used on a variety of cell phones, cell phones cannot use the automatic lane change and speed reduction and so on, only to provide warning effect, hope to eventually achieve full intelligent car distance speed management. REFERENCES [1] Cheng-Kai Luo, “Signal Acquisition and Tracking of GPS Positioning System”, Master thesis, National Kaohsiung University of Science and Technology, July 2017. [2] Kun-Hao Yang, “GPS/Radar based Adaptive Cruise Firetruck Control System”, Master thesis, Da-Yeh University, 2010. [3] Yue-Cin Jhuo, “Development and scenario of Internet of Ambulance Things in 2030 - an evidence of Smart City and IOT based Vehicle”, Master thesis, National Yang Ming Chiao Tung Fig. 7. An example of ambulance or firetruck from behind. University, 2012. 12 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Automatic Randomization in Assignment Question Design Yu-Jia Zhai, Thanasis Theodorou of study, year of entry, module code, etc. Abstract— Constantly emerging new technologies have a significant impact on higher education – both positive and negative. E-assignments and online exams are commonly adopted at universities. The randomization of exam questions can be a way to prevent cheating and plagiarism. However, it can also bring the heavy working load to teachers, especially for teaching in big Fig. 1. System architecture of automatic exam paper generation system. classes. The module leader would like to share an automatic randomization method for exam question design, which makes the The data in input file in Fig 2 can be imported into Python assessment procedure transparent, flexible, and most importantly reproduceable. The method is based on Python language and its program, and the Python program would generate the formatted modules for scientific computing, and it can cover the assignment questions that have randomize parameters. design in the modules, such as programming language, electric circuits, control systems, signal processing, mechatronics, robotics, etc. Index Terms— Engineering Education, Exam Automation, Python Programming, E-Learning, Teaching Technology. I. INTRODUCTION Examination papers in higher education is usually done manually by the academics. The academics design the exam papers according to their knowledge, the nature of modules and Fig. 2. Input file design a format given by their departments. Despite the high credit of the questions, there are still some shortcomings. Human factors It can be seen in Fig. 3 that the output files have two parts. First determine the quality of papers, for example, the skills on word processors and drawing software [1]. Especially, when facing a part is for students, which has only assignment/exam questions large class, the design of assignments that are based on information without solution. This part can be given to students laboratory sessions can be challenging. Teachers need to spend to distribute exam information. a lot of time and energy in composing examination papers, which have the randomized parameters for the same type of questions. With the use of computers, automatic generation of assignment/exam is an important measure for achieving the reliable assessment with affordable efforts. Automatic generation of assignment/exam system can collect the student information and generate randomized questions according to students identify number [2] [3]. Then, it can produce the assignment/exam for students and instructors separately. This can save the marker's time when different students have different expected results because the question for each student is randomized. . Fig. 3. Output file design II. PROCESS DESIGN The second part of output file has all the assignment/exam Fig. 1 shows a system architecture of automatic exam paper information including the solution and marking criteria. This generation system. The student information is given as a part can be used by instructor and marker to perform marking database that include student name, identity number, program when students’ reports were collected. 13 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 III. IMPLEMENTATION In this research, we target the automatic generation of assignment/exam papers for control engineering course. Therefore, a typical question on 1 degree of freedom vibration system is chosen, as shown in Fig. 4 . Fig. 4. 1-degree-of-freedom vibration system To generate assignment/exam paper for this type of questions, Fig. 7. Result of assignment/exam automatic generation 1. the system should have the modules in Fig. 5. Fig. 5. Modules for question design in dynamics In this research, the Python modules in Fig. 6 are adopted to realize the functions required [4] [5]. For each module, the function is provided. The results are shown in Fig. 7 & 8. Fig. 8. Result of assignment/exam automatic generation 2. IV. CONCLUSIONS The system's major characteristics are openness, convenience flexibility, and reproducibility. The system allows the teachers to act according to their demands and design each kind of test question quickly. The system can support marking process with solution information. The automatic paper generation system is a complex systematic project developed through the analysis of examinations in higher education. This paper describes the features and implementation of control system design module. Fig. 6. Python Modules in Implementation. It is possible to extend the idea to other engineering modules with various Python library. 14 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Finally, to practice the implementation of such automation implementation of an automatic examination paper system, the university might need to evaluate the effectiveness generation system, Mathematical and Computer of such systems by a committee. Modelling 51 (2010) 1339-1342. [2] Binghua Chen, B/S Model of Building a Test Server, vol. 08, China's Information Technology Education, 2008, p.87. [3] Gui Wang, JSP-Based Platform for Remote Examination of The Design and Implementation, vol. 10, Mobile office, 2008, p.41. [4] R. Johnson, J. Hoeller, Expert One-on-one J2EE Development Without EJB[M], Wiley Publishing Inc., Indianapolis, 2004. [5] Design and Analysis of Experiments with randomizr, Alexander Coppock, https://cran.r- REFERENCES project.org/web/packages/randomizr/vignettes/randomizr [1] Guang Cen, Yuxiao Dong, Wanlin Gao, Lina Yu, Simon _vignette.html. See, Qing Wang, Ying Yang, Hongbiao Jiang, A 15 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Distilling One-Stage Object Detection Network via Decoupling Foreground and Background Features San-Hao Tsai, Shwu-Huey Yen*, and Hui-Jen Lin Abstract: To deploy an object detection system into an application, the size of the system is one of key issues. We distill a teacher’s knowledge into our small scale network, where the teacher is a full-scale architecture with good performance. In addition to KL divergence loss, we propose a cosine similarity loss on foreground features to encourage student to learn the feature direction of teacher’s. This leads an efficient and robust learning from teacher model. We also propose an adaptive learning criteria which makes student model learns from teacher only when teacher has a better performance than student’s. The proposed student model has an improvement of 34.85% on ResNet34 and 39.67% on ResNet18 when Teacher model is on ResNet50. (a) Index Terms— Knowledge Distillation, Decoupling Features. I. INTRODUCTION Knowledge distillation is a popular method for compressing convolutional networks. The main idea is to use a well- performing large-scale trained network (teacher) to guide a small-scale network (student) during training. By transferring the knowledge of the feature maps, prediction results, and other information of the large-scale teacher network, the small-scale student network can learn better. Therefore, under the same computational demands, performance of a student network trained with knowledge distillation is better than that of the small-scale network trained without [1], [2]. In recent work of knowledge distillation in classification, KL divergence loss on the predicted probabilities between teacher and student is commonly used as a training guide of the student network. However, we found that it is possible that two feature vectors with very different directions but, after Softmax, they have a very small KL divergence loss. Inspired by the above observation, we propose to decouple features into background and foreground, and apply two constraints on them: Cosine-similarity loss to encourage the similar direction on foreground feature vectors and KL (b) divergence loss on background to guide student learning the prediction results from teacher. We also propose an adaptive Fig. 1. (a) The architecture of one-stage object detector RetinaNet which is learning strategy that student learns from teacher only when used as our base architecture. Teacher network & student network are Retina teacher performs better than student does. network with large-scale & small-scale ResNet. In (b), for each level of FPN features from teacher and student, their difference is first measured (purple box with pink border). Then, in classification subnet, four losses are imposed; in bounding box regression subnet, two more losses are imposed. Those purple Manuscript received Oct. 2022. boxes indicate the losses used during training. All authors are with the Department of Computer Science and Information Engineering, Tamkang University, Taipei, Taiwan, R.O.C. (email: a0983367764@gmail.com, *105390@mail.tku.edu.tw, 086204@ mail.tku.edu.tw). 16 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 II. SYSTEM ARCHITECTURE 1 𝑁𝑁−1 We use RetinaNet [3] as the basic one-stage object detection ℒ𝑘𝑘𝑘𝑘_𝑏𝑏𝑓𝑓 = ∗ �𝑖𝑖=0 𝐾𝐾𝐿𝐿(𝐶𝐶𝑠𝑠𝑠𝑠𝑠𝑠(𝑇𝑇(𝑛𝑛𝑖𝑖 )), 𝐶𝐶𝑠𝑠𝑠𝑠𝑠𝑠(𝑆𝑆(𝑛𝑛𝑖𝑖 ))), (3) 𝑁𝑁 network. As shown on Fig. 1(a), Retina uses ResNet as the feature extractor, which outputs features from block 3, 4, and 5 where N is the number of negative anchor box, 𝐶𝐶𝑠𝑠𝑠𝑠𝑠𝑠(𝑇𝑇(𝑛𝑛𝑖𝑖 )) to Feature Pyramid Network (FPN). FPN is a convolutional is a distribution obtained from softmax operation executed on network with a top-down pathway which generates different the classification feature of 𝑖𝑖 th negative anchor box of teacher level features P3 ~ P7 [4]. Each level of Pi laterally connects to net. 𝐶𝐶𝑠𝑠𝑠𝑠𝑠𝑠(𝑆𝑆(𝑛𝑛𝑖𝑖 )) is the same except it is for student network. classification subnet and bounding box regression subnet for The CE loss with GT is evaluate on every positive and negative detecting objects at a different scale. In knowledge distillation anchor box [7]. We use focal loss to alleviate the imbalance learning scheme, teacher net uses a Retina network with a large- problem of foreground-background classes as suggested in [3]. scale ResNet for better features extraction while student net In box regression subnet, there are two smooth L1 prediction also is a Retina network but with a smaller ResNet for losses with respect to teacher’s and ground truth’s. These losses compactness. In the following study, teacher network uses are indicated by purple boxes on Fig.1(c) and is given below: ResNet50 and student networks uses ResNet34 (and later also uses ResNet18). Fig.1(b) depict for each level of FPN features 1 𝑃𝑃−1 from teacher/student networks are fed into classification subnet ℒ𝑏𝑏𝑐𝑐𝑏𝑏_𝑁𝑁 = ∗ �𝑖𝑖=0 𝑅𝑅𝑝𝑝𝑅𝑅𝐿𝐿𝐶𝐶𝐶𝐶𝐶𝐶(𝑁𝑁(𝑏𝑏𝑖𝑖 )), 𝑆𝑆(𝑏𝑏𝑖𝑖 )), (4) 𝑃𝑃 and box regression subnet for object detection, respectively. Since student network cannot extract advanced features as where 𝑁𝑁 ∈ {𝑇𝑇, 𝐺𝐺𝑇𝑇} indicates this loss is measured between teacher does, its performance is worse than teacher’s. To teacher and student, or GT and student, P is the number of improve student’s effectiveness, we design several training positive anchor boxes, and RegLoss is the regression loss on losses to guide student learning from teacher. bounding boxes which is widely used in object detection First, during training, when FPN features are extracted from network [7]. teacher network and student network, the differences between them is measured. Because the student network can perform as Concluding losses mentioned above, we divide losses into well as the teacher only when the student can extract good ℒ𝐺𝐺𝑇𝑇 and ℒ 𝑇𝑇𝑓𝑓𝑓𝑓𝑐𝑐ℎ𝑓𝑓𝑓𝑓 which are losses that student compare with features like the teacher. In classification subnet, there are three GT and Teacher, respectively. types of training losses: (1) Penultimate feature difference [5], ℒ 𝑇𝑇𝑓𝑓𝑓𝑓𝑐𝑐ℎ𝑓𝑓𝑓𝑓 = ℒ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓_𝑓𝑓𝑓𝑓𝑛𝑛 + ℒ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓_𝑓𝑓𝑓𝑓𝑛𝑛𝑓𝑓𝑘𝑘𝑓𝑓 + λ𝑐𝑐𝑐𝑐𝑐𝑐 ℒ𝑐𝑐𝑐𝑐𝑐𝑐_𝑓𝑓𝑓𝑓 [6]; (2) Cosine similarity loss on foreground features; (3) KL +λ𝑘𝑘𝑘𝑘 ℒ𝑘𝑘𝑘𝑘_𝑏𝑏𝑓𝑓 + ℒ𝑏𝑏𝑐𝑐𝑏𝑏_𝑓𝑓 , (5) divergence loss and cross entropy (CE) loss on prediction results. These losses are indicated by purple boxes shown on Fig.1(b). CE loss is calculated with the ground truth label. The rest of losses (four purple boxes) are evaluated between student ℒ𝐺𝐺𝑇𝑇 = ℒ𝐶𝐶𝐶𝐶 + ℒ𝑏𝑏𝑐𝑐𝑏𝑏_𝐺𝐺𝑇𝑇 , (6) and teacher (S & T). where λ𝑐𝑐𝑐𝑐𝑐𝑐 and λ𝑘𝑘𝑘𝑘 are weights to balance cosine foreground The feature loss is calculated as follows and KL background losses. The ratio is 1 : 6 in our experiments. 𝐿𝐿2�∅𝑛𝑛𝑖𝑖 (𝑇𝑇)−∅𝑛𝑛𝑖𝑖 (𝑆𝑆)� Finally, we propose an adaptive learning strategy by a 1 ℒ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓_𝑛𝑛 = ∗ 𝐿𝐿−1 ∑𝑖𝑖=0 , (1) boolean function δ such that the overall losses in training the 𝐿𝐿 ℎ𝑖𝑖 ∗𝑤𝑤𝑖𝑖 ∗𝑐𝑐𝑖𝑖 student network is given below where 𝑛𝑛 ∈ {𝑓𝑓𝑓𝑓𝑛𝑛, 𝑓𝑓𝑝𝑝𝑛𝑛𝑝𝑝𝑝𝑝𝑝𝑝} indicates the feature loss is evaluated ℒ 𝑇𝑇𝑐𝑐𝑓𝑓𝑓𝑓𝑘𝑘 = ℒ𝐺𝐺𝑇𝑇 + δ�ℒ𝐺𝐺𝑇𝑇 (𝑆𝑆) − ℒ𝐺𝐺𝑇𝑇 (𝑇𝑇)�ℒ 𝑇𝑇𝑓𝑓𝑓𝑓𝑐𝑐ℎ𝑓𝑓𝑓𝑓 , (7) on fpn feature or penultimate feature, 𝐿𝐿 is the number of layers, ∅𝑛𝑛𝑖𝑖 (𝑇𝑇) are 𝑖𝑖 th layer of the fpn/penultimate features of the where δ(𝑠𝑠) = 1 only when x is positive, ℒ𝐺𝐺𝑇𝑇 (∙) is the loss when taking the prediction result of student/teacher network with the teacher network (T), and L2(⋅) is the Euclidean norm. These two GT. The δ function makes sure student learns from teacher only feature losses are for different tasks. FPN feature loss is designed for training classification subnet as well as the box when teacher has better performance than student does. regression subnet, while penultimate feature loss is only for the III. EXPERIMENTS classification subnet. We train the proposed student network by COCO2017 The foreground cosine similarity loss is shown as dataset and Adam optimizer. From the previous section, we can observe that the proposed improvement of student learning is 1 𝑃𝑃−1 ℒ𝑐𝑐𝑐𝑐𝑐𝑐_𝑓𝑓𝑓𝑓 = 1 − ∗ �𝑖𝑖=0 𝐶𝐶𝐶𝐶𝐶𝐶(𝑇𝑇(𝑓𝑓𝑖𝑖 ), 𝑆𝑆(𝑓𝑓𝑖𝑖 )), (2) focusing on classification subnet. Therefore, the following 𝑃𝑃 experiments also concern only the classification architecture. where P is the number of positive anchor box, 𝑇𝑇(𝑓𝑓𝑖𝑖 ), 𝑆𝑆(𝑓𝑓𝑖𝑖 ) are We define the Base model of student classification net to be classification feature on the 𝑖𝑖th positive anchor box from teacher equipped with and student respectively. Cos(⋅,⋅) is the cosine of the angle Feature losses: FPN features and Penultimate features. between the two input vectors. KL loss: in addition to kl_bg, we also add kl_fg. The background KL divergence loss is shown as Cos loss: in addition to cos_fg, we also add cos_bg. 17 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 First, we would explore the penultimate feature. Three on ResNet50. Those figures affirm our method can greatly student nets based on the above-mentioned Base model are enhance student’s learning. trained. However, they are different in the penultimate feature loss. As indicated in Table 1, the model with all layers Teacher (ResNet 50) mAP = 34.7 including penultimate layer has the best performance. mAP of Baseline Our method (gained) Student (ResNet 34) 19.8 26.7 (34.85%) All layer but penultimate With penultimate mAP* Student (ResNet 18) 18.4 25.7 (39.67%) V V 11.2 Table 4. The improvement of student network (trained 50 epochs). -- V 11.1 V -- 10.6 IV. CONCLUSIONS Table 1. The difference on penultimate layer. The feature differences is In this research we proposed a distilling object detection evaluated on every layer but with or without penultimate layer (* only trained network which learns from a full-scale teacher network. We 20 epochs) (-- indicates not used) demonstrated that by imposing a feature direction constraint on Next, we explore the difference of KL loss and Cosine loss foreground can result a better learning with an increase of on foreground or background. As indicated in Table 2, we 21.21% mAP. A simple criteria on whether student should learn found out when those models use Cos_fg always have a better from teacher during the training process also add an additional performance comparing to the one does not use. Also, when increase 13.64% mAP. model does not use Kl_fg always has better performance than it To recap, we demonstrated that a Retina model with does use. Finally, the model uses both Cos_fg and Kl_bg has ResNet18 or ResNet34 as feature extractor, under the same the best performance. inference computation cost, can improve mAP by about 35%~40% if trained by the proposed distillation training algorithm. In the future, we would like to explore a soft learning kl_fg kl_bg Cos_fg Cos_bg mAP* scheme instead of hard decision used currently. And, we will -- V V V 23.7 explore more on the bounding box regression subnet, which has V V -- V 21.2 not received the same attention as the classification subnet does. V V V -- 21.9 -- V V -- 24.0 REFERENCES Table 2. The difference on KL & Cos loss on foreground or background. penultimate layer. (*trained 50 epochs) (-- indicates not used) [1] Hinton, G., Vinyals, O., & Dean, J. (2014). Distilling the Then, we would like to know how the adaptive learning knowledge in a neural network. NIPS. strategy works. In Table 3, it clearly shows that the model gains [2] Romero, A., Ballas, N., E.Kahou, S., Chassang A., Gatta, significantly when adaptive learning strategy is used. And, it C., & Bengio, Y. (2015) FitNets: Hints for thin deep nets. helps to detect object disregard the size of the object. ICLR. [3] T.-Y, Lin., Goyal, P., Girshick, R., K, He., & Dollar, P. (2017). Focal loss for dense object detection. ICCV. Adaptive mAP APs APm APl [4] T.-Y. Lin., Dollar, P., Girshick, R., K, He., Hariharan, B., -- 24.0 10.5 24.3 35.9 and Belongie, S. (2017) Feature pyramid networks for object detection. CVPR. V 26.7 12.2 27.2 39.4 [5] Guo, J., Han, K., Wang, Y., Wu, H., Chen, X., Xu C., & Table 3. The adaptive learning strategy. (trained 50 epochs) Xu, C. (2021). Distilling object detectors via decoupled To sum up the effectiveness of the proposed student network features. CVPR. training scheme, we compare baseline student models (trained [6] Wang, G.-H., Wang, Y., Ge, & Wu, J. (2021). Distilling with ℒ𝐺𝐺𝑇𝑇 only) and the improved model trained by our method. knowledge by mimicking features. IEEE TPAMI. In Table 4, the proposed student model has an improvement of [7] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., 34.85% on ResNet34 and 39.67% on ResNet18 when teacher model is Fu, C.-Y., & Berg, A.C. (2016). SSD: Single shot multibox detector. ECCV, Springer, 21-37. 18 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Image Outpainting Based On Attention Model Wei-Chien Tai, Shwu-Huey Yen*, and Yihjia Tsai illustrated edge information is helpful in image restoration since Abstract— Along the advanced progresses on deep neural edge guides the overall structure of the images. networks, there are many impressive results on image inpainting. Attention module is one of the popular data processing Several research works have tried to transfer successful experiences into image outpainting. Contextual attention net is one methods for machine learning. It was initially widely used in of the popular architectural units being applied to inpainting. We natural language processing (NLP), and was later extended to argue that it may be not suitable when embedded in an outpainting different machine learning such as speech recognition or image network. Instead, we adopt SEnet for it has global receptive field processing. The attention module is also used in image and channel-wise feature recalibration. This is very helpful for image outpainting. We also propose a non-fix local discriminator inpainting [19]. It utilizes spatial attention which takes mechanism to decide whether a randomly select partial image is a surrounding image features as references to restore missing real one. By ‘randomness’, the generator can produce a more regions. However, in outpainting, the missing region (or to-be- realistic result. The experimental results are satisfactory and expanded region) is large and far from the existing content. compatible to those on existing state-of-the-arts methods. Contextual attention [19] may be not suitable, we need an Index Terms— Image Outpainting, Image Inpainting, Non-Fix attention module equipped with a large receptive field. Local Discriminator, Attention Module, Squeeze Excitation Network (SENet). Inspired by the above-mentioned observation, we propose a sequential three stages framework to generate semantically coherent extended images. The network consists of an edge map generator, a coarse image generator, and a refinement I. INTRODUCTION generator. Each stage will be combined with the output of the Image restoration in computer vision has a wide range of previous stage to create the final output. In this paper, our applications, including image super-resolution, image technical contributions are: inpainting, and image outpainting. In particular, image We use the structure of SENet as the attention module outpainting technique is very useful when visual media is since it has a global receptive field. exhibited among different devices. Its main purpose is to We advocate the use of non-fixed local discriminators predict the unknown area beyond the boundary based on the in our framework to generate more realistic images. existing content of the image. In contrast to inpainting, image outpainting only has information from one boundary of the image to be expanded. Therefore, it becomes very challenging due to the limited information available. The traditional methods of image outpainting are based on diffusion methods and patch-based methods [1]-[4]. With the rapid development of deep learning, many research uses convolutional neural networks (CNN) and generative adversarial networks (GAN) [5]. GAN was initially proposed to generate images from random one-dimensional input, but after the first use of GAN to repair the image [6], the GAN image outpainting methods have become popular [7]-[9]. Fig. 1. The proposed network consists of three modules. The first stage is an Edge has been an important image feature in various edge map generator. Second stage is a coarse generator. It takes the output of applications. Recently, edge information is adopted in deep the first stage as a conditional input. The third stage is a refinement generator. network for it provides the hints of object boundaries [10]-[13] or information for semantic segmentation or instance II. SYSTEM ARCHITECTURE segmentation [14], [15]. As in [16]-[18], their work also We propose an image outpainting network consisting of three stages, an edge generator module, a coarse generator module, and a refinement generator module (Fig. 1), each of Manuscript received Oct 2022. which can be executed independently. Among them, the first All authors are with the Department of Computer Science and Information and the last stages, the edge generator 𝐺𝐺1 and the refinement Engineering, Tamkang University, Taipei, Taiwan, R.O.C. (email: mark8759@gmail.com, *105390@mail.tku.edu.tw, jaiclab.tku@ gmail.com). generator 𝐺𝐺3 respectively, are GANs. Since edge maps 19 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 determine image structure and texture details, and refinement as shown in Fig. 3. The edge map generated by using a non- yields the final result, both can leverage the well-known fixed local discriminator looks more complete and natural. properties of GAN to create realistic edge-maps/images. In the Coarse Generator. The generator 𝐺𝐺2 at this stage second stage, there is only one generator 𝐺𝐺2 without generates a coarse image from the input image. The architecture discriminator because the produced coarse content serves as an adopts an autoencoder structure similar to the coarse network intermedia image only. In this paper, we aim to expand 1/3 of proposed in [20], but additionally incorporates residual the input image (towards right or left). For convenience, the learning in the bottleneck as shown in Fig. 4. The focus is to following explanation adopts right outpainting. produce an intermediate image to maintain the overall semantic Edge Generator. The edge in the network determines the coherence, without emphasizing structure and details. layout of the image content. With a precise edge map guidance, We concatenate the reconstructed E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 , Ĩ and M as subsequent learnings can focus on color, details in the texture, input, and the coarse output is represented as O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 , as shown etc. The edge generator 𝐺𝐺1 (Fig.2) is a GAN network similar to in (3). Similarly, as in formula (2), O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 = O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 ⊙ M + ̃I. the architecture proposed by Nazeri et al. [16], which proves The output O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 of this rough synthesizer 𝐺𝐺2 will be used as that putting the edge graph together as a condition in the input of the next stage. generator can generate higher quality images. The second and third stages improve the quality of the predicted image by using O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 = 𝐺𝐺2 �Ĩ, E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 , M�. (3) the predicted edge map to obtain the shape of the feature. Fig. 2. Edge generator network. Colored rectangles represent feature maps of different layers. The length of the rectangle represents the relative spatial resolution. The following notations are used: I𝑔𝑔𝑔𝑔 the ground truth image (referred to as I below), E𝑔𝑔𝑔𝑔 the binary edge map of I, and M the mask (with the unknown part to be 1), the masked images ̃I = I𝑔𝑔𝑔𝑔 ⊙ (1 − M) and E �𝑔𝑔𝑔𝑔 = E𝑔𝑔𝑔𝑔 ⊙ (1 − M) (⊙, the Fig. 3. (a) Input image (b) Output image using fixed local discriminator (c) Output image using non-fixed local discriminator (d) Real edge map (e) Real pointwise product). Taking masked images and the mask M as image. input, the edge map generated by the edge generator is represented as E𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = 𝐺𝐺1 �Ĩ, E �𝑔𝑔𝑔𝑔 , M�. (1) Then we combine E �𝑔𝑔𝑔𝑔 and E𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 to be E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 which will be the input for the following stages, shown in (2). �𝑔𝑔𝑔𝑔 . E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 = E𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ⊙ M + E (2) Fig. 4. Coarse generator network. Colored rectangles represent feature maps of different layers. The length of the rectangle represents the relative spatial Next, we use the discriminator to make the predicted resolution. edge map look like the real edge map. Using Ĩ and M as conditions, E𝑔𝑔𝑔𝑔 or E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 are input to the discriminator 𝐷𝐷1 . To Refinement Generator. The final refinement generator ensure the generated image looks plausible to a real one and not 𝐺𝐺3 uses a U-Net architecture with an attention module, SE net, 𝑔𝑔 just to the ground truth, we randomly take 2/3 of the right half to the encoder (Fig. 5), and two discriminators, global 𝐷𝐷3 and of the image, i.e., 1/3 of the size of the entire image, as the input local 𝐷𝐷3𝑙𝑙 . The design is to ensure the refinement module learns of the discriminator 𝐷𝐷1 to decide whether it is a real one. When both the whole and the details. The generator 𝐺𝐺3 takes the a partial image is randomly cropped from E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 as stated predicted edge map E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 of the first stage, the coarse output above, it contains 50%~100% of the predicted edge map. We O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 of the second stage as input. provides the approximate believe that randomly cropped partial images to judge the shape and color of the image. The predicted image generated authenticity can increase the fidelity of the generated images, by the refinement generator is represented as (4). 20 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 O𝑝𝑝𝑝𝑝𝑟𝑟 = 𝐺𝐺3 �O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 , E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 , M�, (4) min max ℒ𝑐𝑐𝑝𝑝𝑎𝑎_1 = min (max �𝔼𝔼�log 𝐷𝐷1 �E𝑔𝑔𝑔𝑔 , ̃I, M�� + 𝐺𝐺1 𝐷𝐷1 𝐺𝐺1 𝐷𝐷1 where O𝑝𝑝𝑝𝑝𝑟𝑟 is the refined extended image, and the final output 𝔼𝔼�log�1 − 𝐷𝐷1 (E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 , Ĩ, M)��� ), (8) image O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 is to restore the original existing real image part. where E𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝 is the composition of E𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = 𝐺𝐺1 �Ĩ, E �𝑔𝑔𝑔𝑔 , M� with the real edge map as indicated in (1), (2). We also use a feature matching loss derived from [23], which is similar to the perceptual loss [24]-[26], but the feature maps for comparison are derived from the 𝐷𝐷1 intermediate layers. The feature matching loss ℒ𝐹𝐹𝐹𝐹 , as shown in (9), encourages 𝐺𝐺1 to learn Fig. 5. Refinement generator network. Colored rectangles represent features similar to real images. feature maps of different layers. The length of the rectangle represents the relative spatial resolution. ℒ𝐹𝐹𝐹𝐹 = 𝔼𝔼 �∑𝐿𝐿𝑓𝑓=1�𝐷𝐷1𝑓𝑓 �E𝑔𝑔𝑔𝑔 � − 𝐷𝐷1𝑓𝑓 �E𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 �� �, (9) 1 where 𝐿𝐿 is the final convolutional layer of the discriminator 𝐷𝐷1 O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 = 𝑂𝑂𝑝𝑝𝑝𝑝𝑟𝑟 ⊙ M + I ⊙ (1 − M). (5) and 𝐷𝐷1𝑓𝑓 is the feature map in the i-th layer of the discriminator. In the discriminator 𝐷𝐷3 , both I𝑔𝑔𝑔𝑔 and O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 are input to Since VGG is trained on the ImageNet dataset over millions decide whether it is a real image. In particular, we design the images for classification, it is not trained for extracting features 𝑔𝑔 global discriminator 𝐷𝐷3 and the local discriminator 𝐷𝐷3𝑙𝑙 . 𝐷𝐷3 𝑔𝑔 from binary edge maps. Instead, we use a much simpler model helps 𝐺𝐺3 learn the image O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 as a whole to be plausible. 𝐷𝐷3𝑙𝑙 , 𝐷𝐷1 for feature extraction. The final loss function of the first module is the same as in 𝐷𝐷1 , helps 𝐺𝐺3 learn the randomly cropped partial image to be plausible, and it can emphasize the seamless effect ℒ𝑝𝑝𝑝𝑝𝑔𝑔𝑝𝑝 = 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎_1 ℒ𝑐𝑐𝑝𝑝𝑎𝑎_1 + 𝜆𝜆𝐹𝐹𝐹𝐹 ℒ𝐹𝐹𝐹𝐹 , (10) on border of the extended region and the real image. λ represents a hyperparameter, in the experiment we choose RSV (Relative Spatial Variant). Generally, 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎,1 = 4 and 𝜆𝜆𝐹𝐹𝐹𝐹 = 10. reconstruction loss is used in training image outpainting models. However, in the case of image outpainting or image inpainting Coarse LOSS. The purpose of the coarse generator 𝐺𝐺2 is of large areas, the effect is often poor because the amount of to roughly generate the extended part. We only use a simple ℓ1 information that can be referenced (compared to the part to be as the reconstruction loss with the regularized RSV weighted generated) is too small. In this case, adding spatial variant mask M𝑤𝑤 which is supervision [19], [21] can effectively improve the performance. Later, RSV (Relative Spatial Variant), an improved version, is ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 = 𝜆𝜆𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 𝔼𝔼 ���O𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 − I𝑔𝑔𝑔𝑔 � ∗ M𝑤𝑤 � �, (11) 1 proposed to regularize the weights of the generated images, where 𝜆𝜆 represents a hyperparameter, in the experiment we which can make the parts far from the original images look choose 𝜆𝜆𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 = 1.2. more realistic [22]. For confidence-driven (CD) loss [21], its formula is shown in (6), Refinement LOSS. For training the refinement generator 𝐺𝐺3 , we also adopt the reconstruction ℓ1 loss with 𝑓𝑓 M𝑤𝑤 � 𝑓𝑓 ) ⊙ M, = (𝑔𝑔 ∗ M (6) RSV weighted mask as � 𝑓𝑓 where 𝑔𝑔 is the normalized Gaussian filter, M = 1 − M + M𝑤𝑤𝑓𝑓−1 0 𝑐𝑐 Lℓ1 = 𝔼𝔼 ���O𝑝𝑝𝑝𝑝𝑟𝑟 − I𝑔𝑔𝑔𝑔 � ∗ M𝑤𝑤 � �. (12) and M𝑤𝑤 = 0. Equation (6) is repeated c times to generate M𝑤𝑤 . 1 The weight matrix used in RSV [22] is: A loss function similar to Contextual Adversarial Loss [22] is 𝑔𝑔 𝑙𝑙 𝑓𝑓 𝑐𝑐−1 𝑐𝑐 used in our study, which is subdivided into ℒ𝑐𝑐𝑝𝑝𝑎𝑎 and ℒ𝑐𝑐𝑝𝑝𝑎𝑎 , M𝑤𝑤 = M𝑤𝑤 �max(M𝑤𝑤 , 𝜖𝜖), (7) global discriminator and regional discriminator, as follows: The parameter c is the number of iterations required to compute min max 𝑓𝑓 ℒ𝑐𝑐𝑝𝑝𝑎𝑎 = the mask, and is set to 9 in our study, the same as in [22]. The 𝐺𝐺3n 𝐷𝐷3 final weighted mask M𝑤𝑤 is used for reconstruction loss in the min �max �𝔼𝔼[log 𝐷𝐷3n (I)] + 𝔼𝔼�log�1 − 𝐷𝐷3n (O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 )����, n Coarse and Refinement stages. 𝐺𝐺3 𝐷𝐷3 Loss Function. The loss functions used in the three modules 𝑛𝑛 ∈ {𝑔𝑔, 𝑙𝑙}, (13) are ℒ𝑝𝑝𝑝𝑝𝑔𝑔𝑝𝑝 of edge generator 𝐺𝐺1 , ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑝𝑝𝑐𝑐𝑝𝑝 of coarse generator 𝐺𝐺2 , The global discriminator will determine whether the input and ℒ𝑝𝑝𝑝𝑝𝑟𝑟 of the refinement generator 𝐺𝐺3 . image is a real one, and the non-fix local discriminator will Edge LOSS. An adversarial loss is used when training randomly select a partial image from right half of the input the edge generator, which is defined as: image and determine whether it is true. Our final adversarial Loss is: 21 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 𝑔𝑔 𝑔𝑔 ℒ𝑐𝑐𝑝𝑝𝑎𝑎_3 = 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎 ℒ𝑐𝑐𝑝𝑝𝑎𝑎 + 𝜆𝜆𝑙𝑙𝑐𝑐𝑝𝑝𝑎𝑎 ℒ𝑐𝑐𝑝𝑝𝑎𝑎 𝑙𝑙 , (14) where λ represents a hyperparameter. In our case, we choose 𝑔𝑔 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎 = 1 and 𝜆𝜆𝑙𝑙𝑐𝑐𝑝𝑝𝑎𝑎 = 15. Perceptual Loss [24] is to input the real image and the predicted image into a pre-trained VGG-19 respectively, and calculate the difference between the activation maps of the two. Our experiments demonstrate that adding Perceptual Loss to the refinement module will improve the quality and stability of Fig. 6. Results of expanding the right 33% by our method. (a) Input image (b) Output the output, which is defined as: Due to hardware limitation and/or unavailable on codes of 𝐿𝐿 the other existing methods, we cannot provide the comparison ℒ𝑝𝑝𝑝𝑝𝑝𝑝 = 𝔼𝔼 �� [�𝜑𝜑𝑓𝑓 �𝑂𝑂𝑝𝑝𝑝𝑝𝑟𝑟 ⊙ 𝑀𝑀� − 𝜑𝜑𝑓𝑓 �I𝑔𝑔𝑔𝑔 ⊙ 𝑀𝑀�� � , (15) with the state-of-the-art methods. We did find one image which 1 𝑓𝑓=1 is shown in Fig. 8, we found only one image that can match the database of [17] which extends the image by 50%. Thus, we where 𝜑𝜑𝑓𝑓 is the activation map of the i-th layer. Style Loss [25] retrain our method for extending 50%. As shown in Fig.8, our is to input the real image and the predicted image into VGG-19 result is also very satisfactory. To demonstrate the proposed respectively, and calculate the difference between the Gram method is also effective on extending 50%, we show the matrices of the activation maps of the two. The loss is defined outpainting results from 128×128 to 256×128 on Fig. 9. as: Although the generated image is different from the real image, 𝐿𝐿 𝜙𝜙 𝜙𝜙 the overall content of the generated content is consistent with ℒ𝑐𝑐𝑔𝑔𝑠𝑠𝑙𝑙𝑝𝑝 = 𝔼𝔼 �� [�𝐺𝐺𝑓𝑓 �𝑂𝑂𝑝𝑝𝑝𝑝𝑟𝑟 ⊙ 𝑀𝑀� − 𝐺𝐺𝑓𝑓 �I𝑔𝑔𝑔𝑔 ⊙ 𝑀𝑀�� � , (16) the natural image in terms of image structure and texture details 1 𝑓𝑓=1 𝜙𝜙 where 𝐺𝐺𝑓𝑓 is the Gram matrix of the activation maps of the i-th layer. Our final refinement loss is expressed as: ℒ𝑝𝑝𝑝𝑝𝑟𝑟 = 𝜆𝜆ℓ1 ℒℓ1 + 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎_3 ℒ𝑐𝑐𝑝𝑝𝑎𝑎_3 + 𝜆𝜆𝑝𝑝𝑝𝑝𝑝𝑝 ℒ𝑝𝑝𝑝𝑝𝑝𝑝 + 𝜆𝜆𝑐𝑐𝑔𝑔𝑠𝑠𝑙𝑙𝑝𝑝 ℒ𝑐𝑐𝑔𝑔𝑠𝑠𝑙𝑙𝑝𝑝 , (17) where λ represents a hyperparameter, and we choose 𝜆𝜆ℓ1 = 1, 𝜆𝜆𝑐𝑐𝑝𝑝𝑎𝑎_3 = 1, 𝜆𝜆𝑐𝑐𝑔𝑔𝑠𝑠𝑙𝑙𝑝𝑝 = 1000, 𝜆𝜆𝑝𝑝𝑝𝑝𝑝𝑝 = 1. III. EXPERIMENTS We use the natural image dataset NS8K [17] which contains more complex and diverse imagery than the original NS6K [27]. We first train the edge generator, coarse generator, and Fig. 7. Results of expanding 33% both on the left and right sides. (a)(c) Input refinement generator separately, and finally concatenate them image, and (b)(d) Output image. together for unified training. A. Experimental results In Fig. 6, the extended images conform to human vision. We hardly notice the seam on the right 1/3 position. The generated content including rocks, ocean view, forest, scene layouts, detailed textures, looks very realistic and natural. The generated images also exhibit coherent content in image structure and semantics. Although our training only generates the right side of the Fig. 8. Compare the results of expanding the right 50%. (a) Pix2Pix [28], (b) original image, we can generate the left side of the input image NSIO [27], (c) BDIE [7], (d) SGSIO [17], (e) our method, (f) GT. (a)~(d) are by flipping the input image vertically. As shown in Fig. 7, we reproduced from [17]. input an image with a size of 144×144, and we can get one 240×144 image by successively extending the content towards the left and the right boundaries. 22 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 patchgan 1:1 patchgan 1:2 our 1:1 our 1:2 PSNR↑ 26.455 26.456 26.458 26.436 SSIM↑ 0.834 0.834 0.835 0.836 FID↓ 3.064 3.008 3.003 2.997 Table 3. patchgan indicates that patchgan is used as the local discriminator, our indicates that the non-fixed local discriminator is used, and the number indicates the weight ratio of the global discriminator and the local discriminator. Finally, we compare the effect of contextual attention [19] and SENet [29] in the refinement generator 𝐺𝐺3 . Contextual attention is widely used in image inpainting which uses convolutional pairing to find the most similar features in the Fig. 9. Results of expanding the right 50% by our method. (a) Input image (b) generated area and the known area. SENet is to learn the Output image (c) Real image. relationship between feature channels, and improve the model performance by calculating the weights of different feature B. Ablation Studies channels. SENet has a large receptive field during the squeeze To find the optimal model, we performed a series of ablation process. In the outpainting task, the generated region does not experiments. For the edge generator in the first stage, we try needs to fit perfectly with the known region. Instead, the different weight ratios of the global discriminator and the non- generated content should be consistent with the known region. fix local discriminator to improve the quality of the generated And a large receptive field will benefit the consistent content. edge map. As shown in Tab. 1, the weight ratios of the global Therefore, the propose system uses SENet [29] in image and the local are set as 1:0, 0:1, 1:1 and 1:2 respectively. We outpainting. As shown in Tab. 4, our model, the model using found that use only the local discriminator when predicting the SENet attention, shows a small improvement in figures. As edge map is the best. shown in Fig. 10, we notice that SENet is more refined in processing details than Contextual is, and the generated 1:0 0:1 1:1 1:2 textures are clearer. PSNR↑ 11.466 11.484 11.374 11.283 SSIM↑ 0.433 0.438 0.43 0.422 Contextual SENet Table 1. The weight ratios of the global discriminator and the regional PSNR↑ 26.482 26.491 discriminator are 1:0, 0:1, 1:1 and 1:2, respectively. SSIM↑ 0.835 0.837 FID↓ 3.068 2.981 For the refinement generator in the third stage, we Table 4. Contextual adopts contextual attention [19], SENet adopts SENet compare the effect on the local discriminator if only take the attention [29]. generated portion (1/3 on the right of the O𝑟𝑟𝑓𝑓𝑓𝑓𝑐𝑐𝑙𝑙 ) versus randomly cropped partial image (also 1/3 of the image but it is on the right half). Let’s call them “fix” and “random”, respectively. We also compare different weighting on global discriminator and the local discriminator between to be 1:1 and 1:5. As shown in Tab. 2, “random” is better than “fix”, and 1:5 is better than 1:1.when the region discriminator adopts random identification, the effect is better than fixed identification, and when the weight ratio is 1:5, it is better than 1:1. Fig. 10. (a) Input image (b) Output image using contextual attention [19] (c) fix 1:1 fix 1:5 random 1:1 random 1:5 Output image using SENet [19] (d) Enlargement of the inset in (b) (e) Enlargement of the inset in (c). PSNR↑ 26.4413 26.2558 26.4581 26.4364 SSIM↑ 0.8343 0.8312 0.8350 0.8352 IV. CONCLUSIONS FID↓ 3.0845 3.0624 3.0029 2.9810 We propose a three stage image outpainting network that can Table 2. fix indicates that the region identified by the local discriminator is extend non-existent content outward from an input image. We fixed, whereas random indicates that it is not fixed. The numbers represent the presented a non-fixed local discriminator to improve the weight ratio of the global discriminator to the local discriminator. naturalness and realism of the generated part, as well as the seamless effect to the border of known region. In addition, we We also compare the non-fixed local discriminator with replace contextual attention with SENet attention, which can patchgan, and the experimental results (see Tab. 3) show that generate finer texture features and better quality images. the effect of using the non-fixed local discriminator is better Extensive experiments demonstrated the effectiveness of our than the latter. 23 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 network not only the satisfactory visual results but salient object detection. IEEE Trans. Image Process., improvement on the metrics of SSIM and FID. 27(1), 121–134. [14] Chen, L. C., Barron, J. T., Papandreou, G., Murphy, K., & Yuille, A. L. (2016). Semantic image segmentation with task-specific edge detection using CNNs and a discriminatively trained domain transform. in Proc. CVPR, REFERENCES 4545–4554. [15] Marmanis, D., Schindler, K., Wegner, J. D., Galliani, S., [1] Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Datcu, M., & Stilla, U. (2018). Classification with an edge: Verdera, J. (2001). Filling-in by joint interpolation of improving semantic image segmentation with boundary vector fields and gray levels. IEEE transactions on image detection. ISPRS J. Photogramm. Remote Sens., 135, processing, 10(8), 1200–1211. 158–172. [2] Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. [16] Nazeri, K., Ng, E., Joseph, T., Qureshi, F., & Ebrahimi, (2000). Image inpainting. In Proceedings of the 27th M. (2019). Edgeconnect: generative image inpainting annual conference on Computer graphics and interactive with adversarial edge learning. arXiv preprint techniques, 417–424. arXiv:1901.00212. [3] Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, [17] Wang, Y., Wei, Y., Qian, X., Zhu, L., & Yang, Y. (2020). D. B. (2009). Patchmatch: a randomized correspondence Sketch-guided scenery image outpainting. arXiv preprint algorithm for structural image editing. ACM Transactions arXiv:2006.09788. on graphics (TOG), 28(3), 24. [18] Xu, S., Liu, D., & Xiong, Z. (2020). E2I: generative [4] Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., & inpainting from edge to image. IEEE Trans. Circuit Syst. Sen, P. (2012). Image melding: combining inconsistent Video Technol. images using patch-based synthesis. ACM Transactions [19] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. on graphics (TOG), 31(4) 82–1. (2018). Generative image inpainting with contextual [5] Ghahramani, Z., Welling, M., Cortes, C., Neil Lawrence, attention. arXiv preprint. D., & Weinberger, Q. K. (2014). Generative adversarial [20] Liu, H., Jiang, B., Xiao, Y., & Yang, C. (2019). Coherent nets. In Proc. Adv. Neural Inf. Process. Syst., 2672–2680. semantic attention for image inpainting. In Proc. ICCV, [6] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & 4170–4179. Efros, A. A. (2016). Context encoders: feature learning by [21] Wang, Y., Tao, X., Qi, X., Shen, X., & Jia, J. (2018). inpainting. In Proceedings of the IEEE Conference on Image inpainting via generative multi-column Computer Vision and Pattern Recognition (CVPR), 2536– convolutional neural networks. In NeurIPS. 2544. [22] Wang, Y., Tao, X., Shen, X., & Jia, J. (2019). Wide- [7] Teterwak, P., Sarna, A., Krishnan, D., A. Maschinot, context semantic image extrapolation. In Proceedings of Belanger, D., Liu, C., & Freeman, W. T. (2019). the IEEE/CVF Conference on Computer Vision and Boundless: generative adversarial networks for image Pattern Recognition, 1399–1408. extension. In Proc. ICCV, 10520–10529. [23] Wang, T. C., Liu, M. Y., Zhu, J. Y., Tao, A., Kautz, J., & [8] Yang, Z., Dong, J., Liu, P., Yang, Y., & Yan, S. (2019). Catanzaro, B. (2018). High-resolution image synthesis Very long natural scenery image prediction by outpainting. and semantic manipulation with conditional gans. In In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Proceedings of the IEEE Conference on Computer Vision 10561–10570. and Pattern Recognition (CVPR), 5. [9] Sabini, M., & Rusak, G. (2018). Painting outside the box: [24] Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual image outpainting with gans. arXiv:1808.08483. [Online]. losses for real-time style transfer and super-resolution. In Available: https://arxiv.org/abs/1808.08483 European Conference on Computer Vision (ECCV), 694– [10] Zhao, J., Liu, J. J., Fan, D. P., Cao, Y., Yang, J., & Cheng, 711. Springer. M. M. (2019). EGNet: edge guidance network for salient [25] Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image object detection. In Proc. ICCV, 8779–8788. style transfer using convolutional neural networks. In [11] Wang, W., Zhao, S., Shen, J., Hoi, S. C. H., & Borji, A. Proceedings of the IEEE Conference on Computer Vision (2019). Salient object detection with pyramid attention and Pattern Recognition (CVPR), 2414–2423. and salient edges. In Proc. CVPR, 1448–1457. [26] Gatys, L., Ecker, A. S., & Bethge, M. (2015). Texture [12] Guan, W., Wang, T., Qi, J., Zhang, L., & Lu, H. (2019). synthesis using convolutional neural networks. In Edge-aware convolution neural network based salient Advances in Neural Information Processing Systems, object detection. IEEE Signal Process. Lett., vol. 26, no. 262–270. 1, pp. 114–118. [27] Yang, Z., Dong, J., Liu, P., Yang, Y., & Yan, S. (2019). [13] Wang, X., Ma, H., Chen, X., & You, S. (2018). Edge Very long natural scenery image prediction by outpainting. preserving and multiscale contextual neural network for In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 10561–10570. 24 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 [28] Sivic, J., Kaneva, B., Torralba, A., Avidan, S., & Freeman, [29] Hu, J., Shen, L., & Sun, G. (2017). Squeeze-and- W. T. (2008). Creating and exploring a large excitation networks. arXiv preprint arXiv:1709.01507. photorealistic virtual space. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops, 1–8. 25 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 An Overview of Adaptive Metadata Prefetching Scheme Wu Chengze, Wan Yamei, Li Wenbin, Kamran Siddique* Abstract— The expensive cost and small capacity of cache has Currently, there are many research directions for prefetching made prefetching a major method to close the performance gap algorithms. Among them are metadata prefetching algorithm between main memory and CPU. Prefetching techniques have increasingly shown their importance as a mean to improve cache and adaptive prefetching algorithm. Metadata is mainly used to hit rates. And Metadata is a kind of significant data, especially in describe data attribute information. It is the bridge between data the information age, where massive internet productions based on and data users. In the metadata prefetching algorithm, metadata it are created and used in daily life. In this paper, we first narrate occupies less space than data, thus reducing the performance the development of prefetching technologies existed to provide a general impression. And two types of prefetching schemes or loss caused by prefetching failure. [3] The adaptive algorithm algorithms, metadata prefetching and adaptive prefetching are considers the blindness of prefetching and improves the introduced from different perspectives. We then compare and prefetching efficiency by adjusting the prefetching length analyze their advantages and present some of their limitations. through a adaptive mechanism. However, adaptive prefetching Then, the further discussion shows how to use the technologies is only effective in the case of continuous data requests, and depending on different factors in the real situation, which may be the size of data. Besides, the self-learning, as a popular theory is prefetching data is largely ineffective in the case of emphasized here for the choice to be combined together with discontinuous user requests for their addresses. Since the prefetching technology. adaptive prefetching algorithm treats multi-user data requests Index Terms—Cache, Prefetching, Metadata Prefetching, as random data requests, it essentially does not prefetch data. Adaptive Prefetching. [5] Thus prefetching performance is limited. In this paper, we will study these two prefetching techniques, analyze the advantages and disadvantages of each, and combine the I. INTRODUCTION advantages of both, and propose research directions for new In Moore's Law predicted, processors are growing prefetch technology based on these two algorithms. rapidly, and performance is increasing. But the access speed of main memory is growing much more slowly. This growing gap II. REVIEW OF EXISTING ALGORITHMS has affected the overall performance of microprocessors. [1] Even though the advent of cache technology can bridge the A. Implementation of Metadata-based Prefetch performance gap between the two, there is still a need to access 1. Prefetching based on semantic analysis data from outside when cache failure occurs. Therefore, cache miss also introduces additional overhead. In order to meet the A prefetching mechanism based on semantic analysis uses demand of today's users for quality of experience, how to origin information to analyze the association rules of element narrow the gap between the processor and the main memory is data and presents a metadata cache prefetching strategy based a question that scholars are constantly thinking about. on origin data. [1] In order to optimize cache performance, prefetching Most of the current research on metadata prefetching techniques have been proposed [2]. Prefetching technology strategies for large-scale storage systems predicts future moves the content to be accessed from memory to Cache in metadata access requests by analyzing historical metadata advance, so that the CPU can get the required content instantly request streams and mining association rules between files and avoid waiting time. However, just as cache can fail, Considering that metadata has rich semantic characteristics, the prefetching also has the problem of prefetching the wrong block origin information of a file is a historical metadata that records of data. Therefore, prefetching data accuracy becomes the main the operation of the file and is a historical archive of file research problem of prefetching algorithms. evolution. This article uses the special metadata of origin information to analyze file associations, thereby improving the performance of metadata services. All authors are with School of Computing and Data Science, This article firstly explains that the set of files being Xiamen University Malaysia, Sepang, Selangor, Malaysia (email: manipulated in the origin information window also has a certain dmt1809291@xmu.edu.my, cst1909152@xmu.edu.my, cst1909140@xmu.edu.my). correlation. And the process set that defines the mutual trigger * Corresponding author: Kamran.Siddique@xmu.edu.my relationship (was triggered by) is called a task, and the life cycle of this task is called an origin information window. Given this, 26 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 it is assumed that the I/O requests in the 2 different origin relationship between multi-user prefetch rate and system load information windows are unrelated. By extracting the origin and prefetch failure cost is analyzed, and the conventional information window, the noise caused by the computational adaptive cache strategy is optimized, and parameters such as correlation between irrelevant tasks is filtered, thereby appropriate prefetch threshold are selected by (Tang et al., improving the accuracy of prefetching. To define the 2013).[4] boundaries of each origin information window, it is necessary Considering the influence of large system load, the cost to obtain the life cycle of each process. The introduction of a function, prefetch rate and prefetch threshold are analyzed, so, threshold max strength for the maximum origin window length as to dynamically adjust the system cache load to obtain the addresses this special case of daemons. minimum prefetch threshold, identify and decompose multi- This article combines the origin information window with user personal information, dynamically adjust the cache the metadata request sequence, the metadata prefetch strategy interval, reduce Cache load, so, as to obtain the highest prefetch ProMP. ProMP can be divided into two steps: (1) According hit rate, which solves the problem of high prefetch failure rate to the metadata request stream in the origin information in the multi-user access shared service system. window, through a time decaying correlation score function, 2. Adaptation of classification methods the correlation score of each pair is counted, and stored in the key-value database. (2) Generate a list of association rules Aiming at the interference of non-memory-intensive based on historical association score statistics, and prefetch programs due to prefetching, (Chen et al., 2021) proposes a future metadata requests. prefetch-aware cache partitioning mechanism based on classification. Using adaptive prefetching control and cache This paper studies the local metadata prefetching on the partitioning technology, it can dynamically adjust the client, and the experiment shows that its cache hit rate is higher aggressiveness of prefetching and reasonably allocate sharing than that of LRU and Nexus 2 algorithms, so the access delay caches. [5] problem of the metadata server is solved to a certain extent. In addition, ProMP needs to store the origin information in the The front-end is responsible for detection, mainly detecting disk. After sacrificing the storage overhead of the disk, it can two types of programs: non-memory-intensive programs and effectively reduce the memory overhead and improve the query prefetch-friendly programs; the back-end is responsible for efficiency of association rules. control and performs corresponding prefetch configuration and cache partitioning schemes according to the results of front-end 2. Prefetching with different locations detection and classification. Metadata prefetched in common distributed file systems is Therefore, the runtime dynamically adjusts the partitioning usually cached on the client side. Because on these systems, strategy by classifying the application and can choose the best- after the client obtains the metadata, it can directly call the RPC performing partitioning scheme based on the stride prefetching. to the storage server to obtain the data. But the process of obtaining metadata and calling RPC in GridDaEn data grid 3. Adaptation of prefetch information table system is completed by DRB as a proxy. So, prefetching The author (Xia et al., 2016) analyzes the change rule of the metadata caching in the client makes no sense and caching in memory access address of the data processed by the X- DSP, the DRB would be the best option. adds a prefetch information table to record the prefetch related According to the characteristics of metadata, user access information, and designs an adaptive prefetch mechanism that mode and the relationship between logical files and directories, changes the step size and quantity in time to adapt to the change (Hong et al., 2009) proposes a same-directory-first metadata of secondary cache failure access. [6] prefetching algorithm, referred to as DHMP (Directory and The idea and mechanism of branch prediction are used to History based Metadata) Prefetching algorithm. Its limitations optimize the design of program prefetching, and the method are the need to set a limited metadata cache size, and a user- based on two-bit saturated counter is used to carry out program mode library limit of rated size. Also pay attention to the prefetching. Based on the prefetch buffer, a historical additional performance overhead for DRB and MDS. [3] information table is added to record prefetch related B. Implementation of Adaptive Prefetching Technology information, and the change rule of memory access failure addresses is analyzed. Based on the local prefetch historical 1. Adaptation of command pre-decomposition information, a method that supports fixed step size and linear The sequential request sequence is obtained by analyzing the change step size is designed as an adaptive prefetching multi-user data request rules and decomposing the random mechanism. At the same time, a confidence mechanism is used request sequence in real time. Based on command pre- to adjust the amount of data prefetching. decomposition and hit rate statistics for multi-user sequential For sequential instructions, prefetching and bit-width requests, self-learning of read prefetch length is realized. The extension techniques are combined to achieve read 27 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 acceleration. For non-sequential instructions, branch cache memory. Origin information and association score information technology is also proposed to reduce the cost of prefetch is stored in a key-value database on disk, and offline update misses. association rules are stored in memory for metadata prefetching. Compared with traditional prefetch policies, the 4. Adaptation of confidence mechanism memory cost is higher, and the memory performance becomes In the report from (Jin et al., 2011), it is proposed through the bottleneck of metadata prefetch. We notice that the experimental tests that the failure rate of the second-level cache efficiency advantage of the metadata prefetch algorithm is greater than that of the first-level cache. At the same time, gradually decreases with the increase of the amount of the access delay caused by secondary cache failure is one of the metadata. In Res data sets, when the metadata is large, it is less main bottlenecks restricting the improvement of CPU efficient than traditional algorithms. performance. [8] As the number of users increases, the average response time for user metadata operations increases. The average response In the prefetching mechanism introduced in this paper, the time of metadata operation under no cache condition is less confidence mechanism of the enhanced step size predictor and affected by the number of users (when the number of users is the idea of step size adaptive adjustment are introduced to less than 200), while the cache only and prefetch cache methods increase the efficiency of cache prefetching. are more affected by the number of users. Through analysis, we draw the following conclusions: (1) limited metadata cache By analyzing the characteristics of cache failure behavior, size. Multiple users share a cache queue, and the metadata is this paper designs a second-level cache prefetching mechanism different for different users (even if the physical files are the with adaptive step size. It avoids the increased hardware design same). As a result, there are too many different metadata in the complexity caused by using the instruction address as an index. workflow, which reduces the cache hit ratio. (2) Limit of user The prefetching mechanism uses an adaptive method, mode library of rated size. It takes a lot of time to store and dynamically adjusts the predictive memory access mode and manage a large user pattern library. In order to ensure that the predictive amount, and can perform prefetching more time of executing the prefetch algorithm is within a certain effectively, and sets the DOI field to avoid the interaction tolerance range, we must set a rating value for the size of user between instruction prefetching and data prefetching. It also pattern library. When the number of users is very large, the uses a confidence mechanism to adjust the number of data space allocated to each user to store the user pattern library is prefetching. [6] very small (that is, the sub-pattern library of each user is very small), so the accuracy of prefetch algorithm cannot be guaranteed. III. FURTHER RESEARCH DIRECTION Thus, the efficiency of the metadata prefetch algorithm is The method of Prefetching relies on the spatial locality of negatively correlated with its size. If you want to take data access to predict the likely occurrence of data requests advantage of this algorithm, you should pay attention to the based on the spatial-to-time concept. It is fetched and cached data size. When the number of metadata is small, metadata prior to access for the user to reduce access latency. However, prefetch has high accuracy, and it is acceptable to exchange prefetch has at least the following disadvantages :(1) affecting sufficient memory space for performance improvement. normal data access load; (2) Incorrect prefetch will reduce the However, when the data size is large, the additional performance of the whole system; (3) It is difficult to achieve performance overhead and memory footprint required by accurate prefetch and complete the prefetch work when metadata prefetching will have a negative effect, and we hope necessary. With the advent of the information age and the to adopt other algorithms to replace it. market demand, it is urgent to integrate the prefetch algorithm The adaptive prefetch strategy considers the blindness of with new technologies. The future research on prefetch prefetch and improves the prefetch efficiency by adjusting the technology will pay more attention to the challenges brought by prefetch length. The adaptive prefetch algorithm treats multi- the fusion of prefetch and new technologies. user data requests as random data requests and does not prefetch data. Therefore, the prefetch performance is limited. A. Fusion of Adaptive Algorithm and Metadata Prefetch The multi-user access service system is optimized by Initially, Traditional prefetch policies apply to common data constructing an intelligent dynamic prefetch strategy with large capacity. Prefetch errors cause cache waste and optimization system. Among them, the prefetch data is an result in high performance penalty. Therefore, prefetch important factor of whether the optimization system can play a accuracy is very important. For metadata with small capacity, role. Therefore, the combination of dynamic Cache size metadata prefetch can greedily prefetch multiple metadata, adjustment and prefetch length adjustment is selected. effectively improving the hit ratio. But metadata itself requires According to the data scale, the dynamic transformation additional processing and can take a bite out of performance. strategy is consistent with the core logic of the adaptive Meanwhile, the metadata prefetch algorithm adopts the offline algorithm. With the combination of adaptive algorithm and batch processing mode, and the space cost consists of the metadata prefetch, we can dynamically adjust the prefetch association rules between the key-value database and the length and choose whether to use metadata prefetch according 28 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 to it. Metadata prefetch is used when the data size is small. between main memory and CPU. Prefetching techniques have Adaptive prefetch is used when the data size is large, and the increasingly shown their importance as a means to improve data requests are continuous. This adapts to all data sizes and cache hit rates. In this paper, we study prefetching algorithms performs well in all situations. based on metadata prefetching techniques and adaptive B. Fusion of Deep Learning and Metadata Prefetch prefetching techniques and summarize two metadata prefetching strategies and four adaptive prefetching techniques. The key of prefetch technology lies in the prediction of And the new research direction of prefetching technology is memory address, so prefetch technology is actually another proposed based on the existing algorithms for reflection. The application of value prediction technology. The main goal of prefetching technique that adjusts the prefetching length prefetch algorithm is to predict the future with the knowledge according to the adaptive and selectively uses metadata of history, so as to effectively improve the response speed of prefetching according to the prefetching length combines the future query. Its performance indicators mainly include advantage of high hit rate when the metadata quantity is small prefetch accuracy and prefetch hit ratio. The prefetch accuracy and the problem of low efficiency when the metadata quantity indicates the extent to which the prefetch algorithm occupies is large with the adaptive mechanism to avoid. The deep system resources, including computing and storage resources. learning-based metadata prefetching technique, on the other The prefetch hit ratio reflects the result of prefetch. The main hand, analyzes the semantic information of metadata by deep factor affecting the prefetch hit ratio is the file access sequence. learning to improve the prefetching accuracy. Both approaches Compared with natural language analysis, file access make some optimization on the existing prefetching techniques sequence analysis is more difficult. Deep neural network can to better exploit the role of prefetching. In future research, the effectively extract high-dimensional features of input data implementation of these two prefetching techniques will be a through hierarchical learning process, and significantly more in-depth research direction. improve the accuracy of prefetching. Recursive Neural Network (RNN) can model the changes of temporal file access and prefetch data, which can be used to predict user REFERENCES requirements during prefetch. Deep learning can provide a [1] G. J. Wu,C. Hu, A Metadata Prefetching Strategy Based better solution for prefetching. on Provenance Information[J]. Computer Engineering, The origin information of a file is a kind of historical 42(6): 1-6, 2016. metadata that records the operation of the file. It is a historical [2] Y. Sun, J. Liu, D. Ye, H. Zhong, Load balancing archive of the file evolution. The current research on data origin framework for metadata service of distributed file systems. focuses on the collection, storage and query optimization of Ruan Jian Xue Bao/Journal of Software, 27(12):3192 origin information. In the field of storage system, a metadata 3207, 2016. management method in cloud storage based on origin [3] G. Q. Hong, F. Liu, B. Liu, et al., Metadata prefetching information is proposed, which effectively improves the and caching strategy in GridDaEn data grid[J]. Computer performance of metadata retrieval. The above work is only a Engineering and Science, 31(S1):335-339,2009. one-sided study of origin data and does not use the special [4] L. M. Tang, S. X. Xing, T. H. Chen, Adaptive Dynamic metadata of origin information to analyze file relevance, so as Prefetch Strategy in Multi-User Storage[J]. Applications to improve the performance of metadata service. Origin of Electronic Technology, 39(01):128-131, 2013.DOI: information can describe the relationships between processes 10.16157/j.issn.0258-7998.2013.01.036. and files. Using provenance information, you can further dig [5] L. L. Chen, T. Jiao, L. Wang, et al., Classification-based out the correlations between documents. prefetch-aware cache partitioning mechanism[J]. Therefore, the rich semantic information contained in the Intelligent Computers and Applications, 11(06):20- origin information of metadata can be used by deep learning 25,2021. technology to extract high-dimensional features, effectively [6] J. X. Xia, J. Z. Lu, H. Y. Chen, Cache Adaptive Prefetch predict file access sequence, and improve the prefetch Optimization Design of X-DSP[C]//. Proceedings of the efficiency. Prefetching technique combined with deep learning 20th Annual Conference on Computer Engineering and and metadata, excavates in the metadata and the origin of Technology and the 6th Microprocessor Technology information effectively for more information, provide a better Forum.:153-159,2016. solution for the prefetch content classification method, file [7] J. S. Jiang, K. Huang, C. Chen, Y. B. Wang, X. L. Yan, access, storage and query. With the development of the prefetch Design of Flash acceleration controller on chip based on technology, deep learning of meta data prefetching technique prefetch and caching principle[J]. Computer Engineering will play a more important role. and Science ,38(12):2381-2391,2016. [8] Q. Jin, Y. Guo, J. Z. Lu, A Step-Length Adaptive IV. CONCLUSION Secondary Cache Prefetching Mechanism[J]. Computer The expensive cost and small capacity of cache has made Engineering and Applications, 47(29):56-59,2011. prefetching a major method to close the performance gap 29 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Design and Implementation of Single-cycle MIPS Processor Luo Yuanyuan, Qin Chenxi, Zhang Wenyu, Kamran Siddique* Abstract— Logisim is a digital logic circuit design and simulation software that is an open source, free of charge, B. ALU Design based on Logisim - 32-bit adder secondary development, installation-free, easy to use and intuitive. To build an adder using Logisim, it is necessary to first This paper examines how Logisim can be used to design the data determine the functions that the adder implements before path and combine it with Verilog for FPGA design of single cycle building the data path. The functions of the 32-bit ALU CPUs. constructed for this project are: arithmetic addition, subtraction, Index Terms— MIPS, FGPA, Logisim, processor multiplication and division, logical with, or, non, and iso operations, logical left shift, logical right shift, and arithmetic I. INTRODUCTION right shift operations. Based existing research, we conducted a detailed study of How about different functions? The ALU will implement single-cycle CPU design and provided a single-cycle CPU different functions and the function codes are transferred via simplified instruction system with Logisim software, VHDL the pin ALUop. language, and FPGA chip. Through instruction control, time The basic addition unit within the ALU is a one-bit full adder, control, and the system design process, this system can fully which implements the addition of 3 one-bit binary numbers utilize the powerful functions and excellent characteristics of (operand 1, operand 2 and rounding) to obtain the output value FPGA. In addition, the single-cycle CPUs studied and and rounding. developed in this article have the following benefits: One is the ability to increase directional instructions and hardware The one-bit full adder feed module is obtained by entering resources to meet different needs. The other is that developers the truth table in Logisim. The data path is shown in Figure 1. are familiar with it. The third one is the ability to control the CPU. I. LITERATURE REVIEW A. Introduction to Logisim Logisim is an educational tool used in designing digital logic circuits and builds circuit simulations. It has easy-to-learn interfaces, layered circuits, wire bundles, and a large component library. As Java application, it can run on many platforms. It can meet the needs of beginners to design and simulate logic circuits. With this software, users can Fig 1. One-bit Full Adder Feed Module automatically generate circuits using multiple sources provided by the simulation design, circuit input truth table and use tunnels to simplify the circuit while setting component A one-bit full adder consists of a one-bit full adder feed parameters. You can package the circuit. This is useful for unit and a heterodyne gate (the low value of the result of blocks. The software works together to also provide data-path calculation). The data path is shown in Figure 2. error detection and display errors in a set of color. Logisim has the ability to build larger circuits from smaller subcircuits and can draw bundles of wires with a mouse drag. Manuscript received September 25, 2022. * Corresponding Author: Kamran.siddique@xmu.edu.my All authors are with the Department of Information and Communication Technology, Xiamen University Malaysia, Sepang, Malaysia (email: cst1709278@xmu.edu.my, Fig 2. One-Bit Full Adder cst1709304@xmu.edu.my, cst1709504@xmu.edu.my). The four-bit full adder consists of four one-bit full adders, two four-bit inputs, one one-bit input (feed) and one four-bit 30 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 output, one one-bit output (feed) and each of the two four-bit inputs connected to pins A and B of the four one-bit full adders and encapsulated respectively. The data path is shown in Figure 3. Fig 5. ADD Addition C. ALU Design based on Logisim - Logical left shift, logical right shift and arithmetic right shift modules Fig 3. Four-bit Full Adder Logisim comes with three modules that implement the logical left, logical right and arithmetic right shift functions of The 32-bit adder consists of eight four-bit full adders. The the ALU. two data bitwise 32-bit input pins are partially led out to the The data paths of the three shift modules are basically the two 32-bit eight-output separators respectively. The first four- same. The logical left shift module, for example, contains two bit full adder is not considered to be fed and is assigned the inputs with a bit width of 32 bits, where input X is the data value of a constant 0. The data path is shown in Figure 4. input and input Y is the shift input. When deciding on the number of bits to be shifted for the X pin input data, the lower five bits of the Y pin are taken through the separator as the data bit widths of the X and Y input pins are both 32 bits. When designing the right shift module, care should be taken to select the shift type of shifter as the desired logical or arithmetic type. D. ALU Design based on Logisim - SUB Subtraction Module This module uses the previously designed 32-bit adder to take the complement of the subtracted number and add the Fig 4. 32-bit Adder subtracted number to the subtracted number to obtain the result. The module also needs to judge signed overflow and unsigned The addition module is divided into the signed case OF and overflow, the principle and the overflow judgment of the the unsigned case UOF. In the UOF case, only the two input addition module is basically the same, in the unsigned values and the result value are compared separately. If at least overflow judgment directly compare the subtracted number one of the input values is greater than the result, there must be and the result size to determine whether the overflow, signed an overflow, so two comparators can be used to connect the judgment need to use the previously designed "adder signed output result to a single gate, and the output UOF signal is the overflow judgment" module. The data path is shown in presence or absence of an unsigned overflow. In the case of a Figure 6. separate "additive signed overflow judgement" unit is required. The ALU operator consists of 12 function modules with a The inputs X and Y represent the highest bits of the two input four-bit binary data path function code, and the output of the values and S represents the highest bit of the result (here the ALU includes, in addition to the result, an OF and UOF highest bits of the three values are separated out in advance overflow judgment and an equality judgment. In the overflow using a separator and fed into the judgement module). judgement, the additive overflow module and the subtractive A logical truth table is entered into Logisim to obtain the overflow module are set up respectively to output a true value addition signed an overflow judgement unit, which is then after detecting whether the ALU opcode is an additive or wrapped to design the overall addition module. subtractive opcode, and output the overflow signal after taking The overall ADD addition module data path is shown in and matching with the corresponding additive or subtractive Figure 5. module to ensure the accuracy of the overflow signal. 31 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 The data paths of the addition overflow judgment module table, and the logical expression of each control signal is the and subtraction overflow judgment module are only for relationship table, control circuit, command system. It is specific designed accordingly and is not easy to implement as it often has tens to hundreds of instructions. The CPU structure is determined by the executable instructions. In this design, 20 basic instructions are selected from the MIPS32 set. According to the instruction function, these are ALU immediate operation instructions. ADDI, ANDI, ORI, XORI, LUI, and ALU registers. Instructions: ADD, SUB, AND, OR, XOR, Shift instruction: SLL, SRL, SRA, Memory access instruction: LW, SW, Branch instruction: BEQ, BNE, Jump instruction: J, JAL, JR. III. IMPLEMENTATION A. Design of Register Block Considering that there are only five destination register Fig 6. SUB Subtraction Module addresses in the MIPS CPU instruction structure, a register group consisting of 32 32-bit register units (the register value opcodes, different ALU addition and subtraction function is always zero) can satisfy the functional requirements. Each codes are different and connected in different ways. Using register has 5 pins in the register setting unit (provided by Logisim's truth table, the circuit for the corresponding module Logisim) : data input (update register data 0 when clock can be built automatically. triggers), clock, enable (ignore clock input when 0), empty The overall ALU Datapath diagram is shown in Figure 7. (asynchronously empty register unit when 1), and output (output the value of register unit). Each register unit is combined into a register block by a demultiplexer and two data selectors. The solution uses the input terminal to input the constant 1, and access the write enable WE signal at the enable terminal and access the 5-bit binary at the selection pin to select the register unit. Finally, two selectors are used to select the register unit and the output data. The input constant 1 is demultiplexed at the input terminal, and the WE (write enable) signal is connected at the enable terminal, and the 5-bit binary is connected at the select pin to select the register unit. Finally, two selectors are used to select the register unit and the output data. The data path diagram of the register block is shown in the Figure 8. Fig 7. ALU Datapath E. ALU Design based on Logisim - Single MIPS CPU CPU design focuses on data path design and control logic design, and the design content analyzes the function of each instruction, provides the necessary original based on the function of the instruction and interconnects. Includes the ability to explore methods. The required control signal values are summarized by the command design control signals, claiming to reflect the command-control signal relationship Fig. 8 Register Block 32 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 according to the corresponding opcode of the ALU op pin output by the controller and output to the data storage. C. Design of Single-cycle CPU Using Logisim to design the path diagram of a single cycle CPU, the relationship between the pins of each CPU component and the signal transmission can be more intuitively understand. After the completion of data design, we can carry out the processor design based on Verilog. In this experiment, VIVADO is used for design and modelsim simulation. And on the basis of the data path, each module and CPU top-level module are designed separately. Finally, simulate the single instruction to write the file. In this experiment, 16 switches on the development board are used, and 16-bit LED is used for output display. After downloading bitstream files to the development board, CPU can be run. IV. CONCLUSION This paper designs the single-cycle MIPS processor. Logisim simulation platform was used to verify the processor design, and the digital circuit system was extended. This . processor supports automatic and single step operation, Fig. 9 Simulation results FPGA correctly execute the program function stored in the main memory and display the main data stream through the LED digital tube in real time, which is convenient for monitoring B. Design of the Single-cycle CPU data path and debugging. The instruction types of single-cycle MIPS CPU are divided In addition to the innovation of circuit design, this topic is into R, I and J instructions, and those with zero opcode field also a training process that includes the integration of teaching, are R-type instructions. The instruction function can be scientific research and project development ability. In terms of determined by the last 6 bits, and the remaining instructions teaching, I fully integrated the knowledge of the gate circuit, can be uniquely determined by the value of the opcode directly. processor, memory, the instruction system and CPU in the Therefore, MIPS instruction parsing is very simple. disciplines of "Principles of Computer Composition" and In the process of instruction fetching, the memory is "Digital Logic", and at the same time had an understanding and accessed with PC as the address, and the instruction word is familiarity with the process of scientific research and project put into the memory, because the whole process of value fetch development. operand execution needs to be completed within one clock cycle. Therefore, address buffer register, data buffer register, REFERENCES and instruction register cannot be set. Instead, access the [1] Yang yang. (2011). Design and Implementation of Single memory directly from the PC. And during the execution of the Cycle CPU Based on FPGA. Computer programming instruction. Fetching operands and fetching instructions both skills and maintenance. require access to memory. Therefore, instruction storage and https://doi.org/10.1109/iwbose50093.2020.9050163 data storage are separated to prevent device expropriation. [2] G Babić, "Designing Single-Cycle MIPS Processor" Similarly, in order to prevent the acquisition of the arithmetic [3] Lin, S, "Instruction translation system and method unit, it is also necessary to access the address of the PC to a achieving single-cycle translation of variable-length separate adder to carry out the cumulative operation of the PC. MIPS16 instructions" , US, US6633969 B1, 2003 When the instruction address of PC is output, the 32-bit [4] Ahmadi, A. , & Mirzaee, R. F, "Mips-core application address is separated into RS, RT, RD, OP, FUNT, IATE and specific instruction-set processor for idea cryptography - other fields through an analysis module (the separator can be comparison between single-cycle and multi-cycle used). Transmitted to the register heap and corresponding pins architectures", 2019 of the controller, the 16-bit immediate number is extended [5] Petrescu, I. , & Petrescu, A, "A VARIABLE SINGLE- through the bit extension module and input to the ALU heap CYCLE INSTRUCTION DATAPATH SIMPLE MIPS PROCESSOR",15th International Conference on Control data output and then transmitted to the ALU, ALU is calculated Systems and Computer Science(CSCS-15), vol.2, Quadrilogic S.R.L. Bucharest,2005 33 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INTERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 [6] Reaz, M. , Islam, M. S. , & Sulaiman, M. S, "A single clock cycle MIPS RISC processor design using VHDL" ,IEEE International Conference on Semiconductor Electronics. IEEE, 2002 [7] Zhiying Wang, Xingshe Zhou Yuan spring breeze, such as computer professional students ability training system and curriculum system set up research [J]. Computer education, 2013 (9) : 1-6 [8] Jiming Wu, Biqing Zeng a highly efficient CPU design method and its application in the course of computer constitute principle [J]. Laboratory research and exploration, 2018 (9) : 147-153. [9] Wu Jie, Qiao Mi, Chunchi Chang. MIPS system in northbridge FPGA design [J]. Small micro computer system, 2004, 25 (11) : 2028-2031. 34 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 Lightweight Rotating Target Detection Model Based on Feature-Aligned Pyramidal Networks Shicheng Liao, Hua Zhong, Wenchong Wu, and Yungang Zhang* Abstract—Most of the current rotating target detection tasks. In the aerial remote sensing image rotating target models choose a deeper backbone feature extraction network detection task, the real-time performance of the model also with a large number of parameters to extract the image has high requirements. At this stage, several methods to features, the FPN structure is usually used for feature improve the real-time performance of the model mainly enhancement of the extracted features, and finally for target detection, such models can achieve high detection accuracy, include compressing the trained model, lightweight however they are not suitable for the application scenarios network model design, accelerating the speed of with high real-time requirements, such as edge computing. To convolutional operations, etc. In this paper, we choose address the above issues, in this paper, a single-stage rotating FCOSR-S [1], a single-stage unanchored frame rotating target detection model FCOSR-S based on lightweight target detection model based on a lightweight backbone network design will be selected as the baseline model, and the feature extraction network ,as the baseline model to carry feature-aligned pyramid network will be combined to out research. The FCOSR-S model proposes an ellipse improve FCOSR-S. Finally, the improved model will be center sampling strategy, a fuzzy label assignment strategy, trained and evaluated on the homemade human-vehicle and a multi-level sampling strategy that are more suitable rotating target detection dataset and the DOTA dataset. The mAP on the two datasets are improved by 2.69 and 0.43 for rotating target detection based on the FCOS [2] model, percentage points, respectively, while maintaining a high and the MobileNetV2 [3] network is selected as the detection speed, therefore the overall performance of the backbone feature extraction network, making the whole model is improved by the proposed method. model a lightweight network with a fairly good detection Index Terms—rotating target detection; lightweight model; accuracy at the same time. feature alignment Although the FCOSR-S model has achieved good detection results, this paper finds that the feature pyramid network (FPN) [4] in the model has the problem of feature misalignment in the process of upsampling feature fusion, I. INTRODUCTION which greatly limits the detection performance of the With deep learning research, computer vision-related model. Thus this paper increases the FCOSR-S model by work has made great progress, among which the task of combining the feature alignment pyramid network (FaPN) target detection based on aerial remote sensing images has [5] to improve the model detection accuracy while also attracted much attention. Aerial remote sensing maintaining the high inference speed of the model. Making images generally have high resolution and often contain a the model more balanced in terms of detection accuracy huge number of targets, among which there are many small and detection speed. Finally, the improved model is trained targets, which makes detection extremely difficult, and at and evaluated on the homemade human-vehicle rotation the same time, the targets are arranged in the images very target detection dataset and DOTA dataset [6], and the messy, so the target orientation information needs to be experimental results are analyzed to verify the predicted when localizing the targets, i.e., the rotating effectiveness of the method in this paper. target detection algorithm is used for target detection, so as to achieve the purpose of accurate target localization. II. RELATED WORK In recent years, some rotating target detection algorithms have achieved good detection results in aerial remote A. Rotating target detection models sensing image target detection tasks, and these models have high detection accuracy higher, but ignore the Most of the target detection models at this stage are problem of model complexity as well as model obtained by improving on the general target detection computation, which is difficult to deploy to end devices in model by first increasing the orientation prediction of the practical application scenarios. Moreover, as the research model, and then improving the model for its deficiencies in progresses, researchers find that the practice of using a various aspects, and evaluating the model on a public huge network to improve the detection accuracy of the dataset to prove the effectiveness of the proposed method. model will lead to problems such as oversized models, not A fine sampling fusion network and a multidimensional easy to deploy and low inference speed, for example, in attention network are designed in SCRDet [7], in which the tasks such as unmanned driving and face recognition, fine sampling fusion network adds a fine sampling module terminal devices often do not have such high before the feature fusion operation of the feature pyramid computational power, so the research of a more lightweight and feeds the output feature map of the fine sampling and high detection accuracy model to better adapt to these module into the multidimensional attention network containing the pixel attention mechanism and the channel All authors are with the School of Information Science, Yunnan Normal attention mechanism, so as to improve the overall University, email: *yungang.zhang@ynnu.edu.cn performance of the model. CADNet [8] proposes global 35 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 and local context networks and designs a spatial and rotating target detection model, and simplifies the model scale-aware attention module to improve the model's structure by removing the center-ness branch in FCOS, in ability to utilize information around the target. R3Det [9] addition to analyzing and improving the sampling strategy provides an in-depth analysis of the rotating frame and label assignment strategy in FCOS, including localization problem in dense scenes ,and a new loss proposing a more suitable ellipse center sampling strategy function is designed to obtain more accurate loss estimates. for the rotating target detection task, a more advanced Retinanet-O [10] improves the original Retinanet into a fuzzy label assignment The effectiveness of the proposed rotating target detection model with a simple structure and method is demonstrated experimentally. a focal loss function, which gives better results in dense target detection tasks ,and the model has a high inference B. Feature alignment methods speed. ATTS-O [11] improves the original ATTS into a rotating target detection model, pointing out that the There are various methods to improve target detection sampling method of samples largely affects the models, and feature alignment methods are one of the key performance of both anchored and unanchored target points to improve the model performance, and this is detection models, and proposes an adaptive sample especially true for rotating target detection models. A new sampling method improves the performance of the model alignment convolution calculation method is proposed in and proves that laying multiple anchor frames on one S2ANet [12] to address the problem of misalignment of feature point is unnecessary. New alignment convolution is localization frames and feature points, and a feature proposed in S2ANet [12], and the feature alignment alignment module is specifically designed to generate module is designed to generate high-quality suggestion high-quality rotating suggestion frames. The adaptive frames, after which the final detection results are obtained feature alignment module in DARDet [15] alleviates the by refining the rotation suggestion frames with the rotation problem of the feature point and locus frame misalignment, detection module. ReDet [13] provides an in-depth analysis and the module is less computationally intensive and does of rotation equivariance, incorporates rotation equivariance not significantly slow down the inference speed of the into the feature extraction network and feature fusion model. These models improve the detection accuracy by network, and proposes a rotation-invariant region of different feature alignment methods, but mainly address interest feature alignment method, which greatly improves the problem of misalignment of localization frames and the performance of the model. Oriented R-CNN [14] feature points, ignoring the problem of feature proposes a candidate frame generation module that misalignment in the FPN used in the model. To address efficiently generates high-quality candidate frames and this problem, this paper will choose the feature alignment then extracts fixed-size features through the rotational pyramid (FaPN) [5] structure to alleviate the problem of region of interest feature alignment module, and finally feature misalignment when fusing the upper and lower uses these features as the input for directed head detection feature maps in the FPN and improve the overall as a way to improve the detection performance of the performance of the model. model. FCOSR [1] improves the FCOS [2] model into a Fig. 1 Overall structure of the model. head part. Where MobileNetV2 is used as the backbone feature extraction network and C1 , C2 , C3 three scale III. THE METHOD PROPOSED IN THIS PAPER feature maps are selected as inputs to the FaPN. After the The approach of this paper is mainly based on the FaPN is inputted to C1 , C2 , C3 , C2 and C3 go through the FCOSR-S model. The overall structure of the model is first introduced, followed by a detailed discussion of the feature selection module for feature enhancement during MobilenNetV2 network and the feature-aligned pyramid upsampling, respectively, before being inputted to the structure part of it, and the advantages of the model design feature alignment module along with the previous scale are elaborated. feature map for feature alignment and feature fusion. Finally, the feature maps at different scales output from the A. Model Introduction FaPN are fed into the detector head of FCOSR-S and localized and classified to obtain the final detection results. The overall structure of the model is shown in Fig. 1. The model can be divided into three parts, which are the B. Lightweight backbone feature extraction network feature extraction network part, the feature alignment pyramid network (FaPN) part, and the FOCSR-S detector 36 INTERNATIONAL JOURNAL OF DESIGN, ANALYSIS AND TOOLS FOR INERGRATED CIRCUITS AND SYSTEMS, VOL. 12, NO. 1, JANUARY 2023 The feature extraction network of the lightweight accuracy of the model in tasks such as image classification rotating target detection model FCOSR-S selected in this and target detection. In order to better understand the paper is the MobileNetV2 network, which is obtained by MobileNetV2 network, this section will focus on the improving the MobileNet network [16]. The principles of deep separable convolution and inverse depth-separable convolution in MobileNet reduces the residual structure in this network. computational effort of the network, but there is the Deeply separable convolution. The depth-separable problem of zero training results of the convolution kernel convolution is designed to reduce the computational effort and the lack of residual design. To address these problems, of the network, so it is improved on the basis of the MobileNetV2 designed the inverse residual module, which standard convolution, as shown in Fig. 2. not only reduces the computation of the model again and makes the network more lightweight, but also improves the Fig. 2. Deeply separable convolution. convolution, in addition, other hyperparameters are additionally designed in MobileNet, which also reduces the From the Fig. 2, it can be seen that the depth-separable computation of the network. convolution changes the k k Cin size convolution kernel Inverse residual module. The inverted residual structure in the standard convolution to a k k 1 convolution kernel, is shown in Fig. 3. For the input channel Cin , the input and then uses the k k 1 convolution kernel to convolve feature map is first up-dimensioned using Cin t each channel of the input feature map to obtain a set of convolutions of size 1 1 , followed by depth-separable feature maps with the same scale as the input feature map, convolutions for convolutional operations, and finally Cout and then uses the 11 Cout convolution kernel to change convolutions for dimensional operations. It is important to the number of channels of the feature map and outputs the note that the Relu6 activation function is used for the first final feature map. In this process, it is not difficult to dimensional increase and the Liner activation function is calculate that the computation of depth-separable applied after the dimensional decrease. The use of the convolution is k k Cin + Cout , and the computation of Liner activation function avoids feature loss, and the standard convolution is k k Cin Cout , thus the design of the residual linkage improves the performance of computation of depth-separable convolution can be the model. introduced as 1 Cout of the computation of standard Fig. 3. Inverse residual structure. 37
Enter the password to open this PDF file:
-
-
-
-
-
-
-
-
-
-
-
-